Create your own Telegram bot with Django on Heroku – Part 3

Header Image showing Django, Telegram, Heroku logos

In the previous part of this series, I explained how to register bots on Telegram, how to configure it and how to validate everything is working.

Today I will explain a bit more on how the HTTP API works, how the JSON data provided by the bots ist structured and I will introduce you to telepot, the Python module of my choice for interacting with Telegram bots using Python.

Continue reading “Create your own Telegram bot with Django on Heroku – Part 3”

Create your own Telegram bot with Django on Heroku – Part 1

Header Image showing Django, Telegram, Heroku logos

Recently, I finished my first Django application ✌. Since it took me some time to find a proper starting point to wrap my head around the basics of Django well enough to actually do something with it (apart from following tutorials, which pretty much all aim for creating a polling app for any reason 😋), I’d like to recap and share that journey by writing this article about that.

I will describe the project, my thoughts and considerations about it and every step which has lead to what the project has become. I will try my best to make this a guide which everybody can follow along if he wants.

Also, I will make this a series, to not end up with an article so long and boring, that even interested readers cannot afford to read it in their spare time.
In this first part of the series, I will explain what this series will be about, mainly.

Continue reading “Create your own Telegram bot with Django on Heroku – Part 1”

Updated Docker image derjudge/confluence

Confluence Logo

Hi everyone!

This is just a short notice to let you know, that I just updated my Docker image derjudge/confluence to provide the most recent version of Confluence (6.10.1) and Java 8 (8u181). Enjoy!

derjudge/confluence is a Docker image which provides Atlassian Confluence in a single “batteries included” Docker container.
Please find additional details on the Docker Hub page of the image and feel free to report any trouble using the bugtracker at Bitbucket.

 

Album Review: Era Capricorn – I Wander Alone

This will be the first music review I have ever done, so please do not roast me too much 👶

Foreword

The reason why I have selected this record to be my primer for this is the fact that I’m not only a close friend to the artists but also I was utterly impressed with the results.

The record is self-produced, -released and -sold by the band’s official shop, only, created solely for this record (so far! 😉)

Even though we are close friends, I (among others) only learned about this project quite late, when large parts of it were made and recorded already in the middle of December 2017. This was because “Era Capricorn” was started as a personal, these two persons only – like project. The two of them did not want any distractions or being blocked by having to align anything with anyone else or have pushing-like “suggestions” in the way of their very own creativity process; which I consider a perfectly understandable decision. In the end, this is their first time record also and no one is immutable and ultimately confident in their mindset in that situation, no matter what distractions, alternate opinions or impediments may appear on that road ahead.
Even though everything was quite final already in December 2017, it took another 4 months (until April 2018) to fine-tune on the details before the first package containing the finished records could be ordered and be shipped by the CD / printing company.

Alex told me, that they wanted to make this first record something which is truly and completely their own minds child without anybody else possibly altering the outcome. This is not only true for the main content (the music), but also for everything else like the cover artwork, the overall design, the ordering process at the printing- and CD company, coordination with GEMA and so on.
Alex, who is also a capable hobby photographer, took the cover shot in the woods of a nature reserve, close to their home place when there has been an unusual amount of snow for that region. Marina also was part of that photo session and supported during the choice of a composition of the image structure and layout. After the shot was taken, the two of them post-processed the image on their own at home, arranged the Capricorn to match that scene, hassled around with the printer’s CMYK requirements, and so on.
Marina even created a unique font for this, instead of just using something prebuild from the internet; all to make this record something as unique as themselves and the music it contains.

When they told me that they are in the final sprint to self-release their first album, I expected to receive some songs as MP3 or some self-burned and Ink-Printer labeled CD. But what they delivered is a professional appearing, full-fledged CD with only the barcodes and labels missing, compared to a label backed release:

The music

Why am I talking so much about how appealing the design of the CD is and how much work was spend on having a nice overall appearance? Why don’t I start with the main content – the music?
Because I’m convinced that often self-releasing bands, especially primers and even more instrumental ones are misjudged in several ways. Instead of accepting the absence of lyrics as an intentional compositional decision, some judge it as “They did not even find a good singer!” or “They seem to have no lyrical skills!“.  If any instrument or arrangement sounds unusual, famous and settled bands are conceded to make use of some artistic freedom and being “refreshing” or similar, while primers are judged to not being experienced or skilled enough to make it successfully sound like the well-established schema from a can.
That’s why I started to point out how much efforts and heart blood has lead to this record before.

As pointed out before, “Era Capricorn” are Alex and Marina, living in Essen (Germany). Both are playing instruments and create music as a hobby. Alex started to learn to play guitar in April 2014. Marina learned to play drums around 2014, but switched to classical piano and keyboards in October 2016.
I think to have this quite short period of learning the instruments in mind, makes the listening experience something even more notable.

Even though Alex is effectively executing most of the hands-on tasks, like operating the DAW-/Photo-software and the Drum computer, the two of them were working hand in hand to achieve their mutual goal.
A perfect example are the drums: Just having the muscle memory and technical skill to operate a computer does not enable you to achieve great drum beats. Also, just knowing how to play drums does not enable you to operate the software necessary to turn your imagination into sound. Joining their forces on this task, just like with nearly all other details during the project, lead to the awesome result this record is.

Having listened to the record for some time now, without any complaisance involved, I’m still sincerely impressed in the tightness and atmosphere they managed to create here. I’m still catching my mind red-handed, searching for some rough edges in details, like passages unaligned to each other, too abruptly ending sequences, cheapish sounding instruments and there like, but nearly always, when I think I found something, when re-hearing that passage something else in the complex composition unveils and proves my ear wrong, because it matches nicely; just with something I was not capable to align my ear to in the first place.

On their official Youtube channel, they released a teaser-medley first, playing a sequence of “Wiedereintritt”, “Remember my name” and “Watch me burn” for ~30 seconds each, starting and ending from somewhere in the middle. This teaser contained some unfinished early previews of the songs; in the meantime this was replaced by a full album teaser:

 

I have to admit that I’m having a hard time to get a feel for a song when presented like this, so I managed to earn shaking heads in disbelief from them when I told them so 😳

“Wiedereintritt” is the first song the band released in full length in January 2018; even before the album was finally recorded completely. I was only able to provide first feedback on this one.
“Wiedereintritt” is a bit special in two ways. First, it contains an example for the only criticism I can track down: There are bells at around 1:40, which “bite” into the ear a bit too sharp, compared to the rest of the arrangement, when listening to the song using headphones in a slightly louder-than-flat – like volume. But that is only half way negative criticism: It also is somewhat positive, since most productions nowadays are flattened down in post-processing to make the songs sound alike everywhere; no matter if you are an audiophile, listening to a record on a central chair in your personal, $50.000 dollars sound-room or some crack-head, listening to the lowest MP3 quality using your $1 headset.
Having these “un-flattened” spikes is something you most probably have only if artists are not bound to align with some money-first – approaches put on them by a label.
The second reason “Wiedereintritt” is highlighted here is that I had a conversation with Alex about the fact that I like to know what the artist’s intention was when they created a song. Even more for instrumental songs, where there is no text, giving you an idea. He told me this:

In the very early phase, the project was planed to contain vocals. Even though this plan was scrapped very soon, “Wiedereintritt” was finished at that moment, already. So were the (now scrubbed) lyrics.
The song is about the re-entry process of a spaceship into earth’s atmosphere; something arresting me very much, personally. It inhabits something sublime and epic, while being something stressfully and dangerous at the same time. This is something we try to address in the first half of the song.
The passage in the middle of the song, when the theme suddenly changes (at around 4:00) reflects a somewhat critical phase during the maneuver; the theme becomes more tragic, all sublime has gone, being replaced by shock and disbelief; even though the tempo keeps pace.
The imaginary story around that scene has even more details, but this is not shared; some parts are left to the imagination on purpose.

The goal of the outro was aimed for ending the song epically, whilst releasing the pressure more and more.

Another interesting side info: Alex is the biggest Transformers fan I know. Not only since the movies from Michael Bay generate hundreds of millions of dollars at the international cinemas box offices, but since his childhood days when the first animated series hit the air. His whole flat is covered with self drawn pictures of epical Transformers scenarios, merchandise, action figures and stuff.
He put a Transformers – themed trilogy onto the record as well:

It’s about “Watch me burn”, “Embers” and “Remember my Name”.

“Watch me burn” themes the last fight of Optimus Prime, when he is defeated by Megatron (e.n.: something which is happening all-the-time, but I think this time the epical scene in “Transformers: Revenge of the Fallen (2009)” is refered to).

“Embers” audializes the following scene, when he (e.n.: Optimus Prime) is dying while being relocated to the headquarters.

“Remember my Name” is something I wanted to finally salute him (e.n.: Optimus Prime) with; a hymn on my childhood lifetime hero.

I had to experience some katharsis from my childhoods trauma on that one … 😉

The whole record is never boring or unpleasant to listen to. Even if there are passages which are not a perfect match to one’s taste, the record keeps promise to dynamically changing all the time. All tracks are diverse a lot and satisfy many of the different preferences out there at the same time.

As Walter Scheurer from powermetal.de already pointed out in his review (German), noone in the formation needs to self-express or show off; everything they do and put together are exclusively sub serving the songs theme and to drive it forward, making it a unique thing you enjoy to listen to.
Personally, I tend to skip most of those longer guitar solos, not sounding good really, but obviously just in there to show off (truth be told: most of the time I’m too lazy to skip, though. But not skip those parts because I’m too lazy does not mean that I enjoy listening to them).
This is not the case on this album even once. I really appreciate that!

To sum this up: This album is a great and outstanding piece of music, where no track needs to be skipped; not even partly. I really enjoy listening to this record; but you need to have a fable for instrumental music.

Process a textfile line by line in Bash

GNU BASH logo

One of the most common tasks when working with Bash-scripts and the Linux shell is text processing like filtering, selection, transforming, …

Often, these texts come from text files like CSV, log files, and so on. If you are an experienced user who is doing this on a daily basis, typing these kinds of command chains often feel like they are coming from “muscle memory” more than from your brain. But most of the time, you need only parts of these lines, like “the 5th to the 7th field” or some regular expressions match; these are usually quite easy to catch using a combination of the well-known tools awk, grep, cut or sed.
But: How to iterate (loop) over each line of a file in Bash and use that value for your processing; like, if you want to preserve only lines, matching a specific pattern or divide the script of a play into separate files per role, for example?

This is not really a hard task. But unlike the formerly mentioned processing for but parts of that lines, doing such things for the whole line is not a very commonly needed thing for my daily work. That’s why this gives me a hard time when I need it in those rare cases because that is not stored in my very own “muscle memory” and it enforces me to crawl through my memory castle and dig for it (or Google for it 😅).

The solution

Sure thing, there are plenty of ways to solve this, including not utilizing Bash in the first place, but some real programming language like Python 🐍. But to me, the following approach has proven itself as the most effective and easy to remember one:

Explanation:

  • IFS=””
    prevents leading/trailing whitespace from being trimmed.
  • -r
    prevents backslash escapes from being interpreted.
  • || [[ -n “$line” ]]
    prevents the last line from being ignored if it doesn’t end with a newline (\n), since read returns a non-zero exit code when it encounters EOF.

Instead of the “echo”-line, you can do whatever you like with the ${line}-variable, of course!

An Example

The following example is using this sketch.txt – file, which contains the text of a Monty Python sketch. It uses the proposed solution to separate its content into two separate files for each of the roles:

When you save this as “sketch-process.sh”, set it’s execution bit and put it into the same folder like formerly mentioned sketch.txt – file, you will end up with another 3 files after you executed “sketch-process.sh”:

  1. garbage_lines.txt
    … containing the “(pause)“-lines
  2. man_lines.txt
    … containing the lines starting with “Man:
  3. otherman_lines.txt
    … containing the lines, starting with “Other Man:

I hope you liked this article and that it turns out being helpful for some! Let me know in the comments ✌

Fix missing Google calendars in Evolution / CalDAV

Google Calendar and Evolution Sync

Me and my wife both use Google calendars to organize our daily schedule. Also, we share these calendars with each other, to see each other’s appointments. This way, we do not clash each other’s plans by accepting appointments in the same, concurrent time slots.

Recently, we found that not all of her calendars were offered to me in Evolution. After some digging, I found the solution and I’m going to explain it in this article.

Since some people in charge (note that I did not mention “we” 😤) decided to switch to Office365, I need to use Evolution PIM on my Linux machine to have a slightly enjoyable Exchange-experience, at least. The other solutions, like DavMail for example, worked basically but proved to be too error-prone and slow for my taste; more often than never, Mails I archived in Thunderbird with DavMail in between showed up again after some Minutes and it took ages for any action.
However, somehow I could not select all of these foreign “shared” calendars, my wife shared with me – they simply were not listed in the select dialogue.

How to solve this

I finally found the solution to this issue here: It turned out that it’s not Evolution’s fault; instead, Google does not advertise (list) these calendars by default. You first have to change this (quite confusing) default setting at the following location:

https://www.google.com/calendar/syncselect

Please mind the lower list on that page; this should be a complete list of calendars shared with you. Select those you want to be able to access in CalDAV based clients (including Evolution) and save your selection. Your changes should be effective, immediately.

I hope this is helpful to some – Please let me know in the comments✌

Fixed my most popular Docker Image

Confluence Logo

Today, I’d like to announce that my most popular Docker Image derjudge/confluence, a batteries-included solution to get Atlassian Confluence up and running with a mature database (PostgreSQL) as storage backend in seconds, has been fixed and updated.

  • It ships with the most recent version of Confluence now, which is 6.8.1.
    The image was not updated since Confluence version 6.0.2 before … sorry for that!
  • PostgreSQL version was updated to 9.6.
    Since Atlassian has decided to finally support this version, I declared it to be the version of choice in my image, too. It was set to be 9.4 before.
  • JAVA version was updated to 1.8.0_162.
    This has been 1.8.0_112 before.
  • Underlying Debian release was updated to “stretch”.
    This was “jessie” before.

I have to admit this had not really received some love, recently … but to my defense: I do not use it for hosting Confluence myself, currently. Not that I do not like to taste my own poison, but the infrastructural environment my hosting is built up in does not need it, currently. So I do not really realize if anything breaks (which was the case with PostgreSQL not launching, recently).
Also, nobody got in touch with me, telling something is wrong; the first note on this issue (PostgreSQL not working) I received by mail on 2018-04-12 at 11:04 CEST (thank you, Michael Bykovski from //SEIBERT/MEDIA!); on 2018-04-13 at 20:13 CEST the fix has been made, the formerly listed updates were applied, a new Image tag was created for this new release and the image was built, successfully.

I wonder a bit why nobody has done before: The Image has 50K pulls (WOW – thank you!! ), the PostgreSQL issue seems to has been in there since December 2016 (!) and both, my E-Mail address and the link to my source repository which has an issues reporting feature are both prominently available to the images Docker Hub page.
Guys: I can only fix things I know of, so:

Please utilize the tools offered to get in touch!

I hope I’ll find the time to push newer versions more often, proactively in the future. If I miss something: Feel free and actually invited, to poke me! 😉

More recent Python in Enterprise Linux like CentOS and RHEL

Tux

This article describes what “Enterprise Linux” is and how to add a more recent version of Python to it than those available in the base package repository.

What is “Enterprise Linux”?

General definition

CentOS and RedHat Enterprise Linux (RHEL) both are counted as one of the so-called “Enterprise Linux” systems. This term is an artificial noun, which has different meanings. In general, this describes Linux distributions, which are targeted at the commercial market thus putting a strong focus on reliability and long lifecycles.
CentOS, RHEL and SUSE Linux Enterprise Server (SLES) usually maintains a release for 10 years; RHEL and SLES even offer extended support contracts for additional years of support. That means these distributions offer at least twice as long support for a version than Ubuntu LTS versions does (which usually is ~5 years).

This kind of distribution’s biggest strength often is also one of their biggest downsides: If you want to have a more recent version of any of the software they contain, you often have bad luck. More recent versions (if any are available at all) usually come from 3rd party repositories. One of the most famous ones for CentOS is EPEL (Extra Packages for Enterprise Linux), which ports a lot of “had expected that to be available” – like packages from Fedora to Enterprise Linux.
But the more repositories you add, the more unpredictable and unreliable the core becomes.

RedHat based definition

But there’s also another meaning: The term “Enterprise Linux” also has established as a term to group distributions, which are based on blueprints from RedHat Enterprise Linux or build alike it. Some sources refer to this kind of distributions like this (like EPEL, for example). This list normally includes (but is not limited to):

RedHat Family Tree
RedHat Family Tree

This does not mean in any way, that other distributions, not based on RHEL, are not enterprise class or ready!

What are Enterprise Linux distributions used for?

Enterprise Linux distributions are often used in large-scale IT environments with several hundred or thousand hosts. In this kind of environments, reproducibility (by orchestration/ automatization), reliability, compatibility and hardened concepts and versions are key aspects.

In large orchestrated IT environments, an often selected choice for a distribution as a base is CentOS since, being an Enterprise Linux distribution, its main focus aims at being a rock-stable “enterprise-class” platform prioritized over delivering the latest upstream versions of software selections. It also aims at being binary compatible to RedHat Enterprise Linux (RHEL) while being free of charge and only community supported.

Talking about CentOS …

 

CentOS logo

I will stick to CentOS here since this is the Enterprise Linux I utilize the most. But since we are talking about “Enterprise Linux” here, the following should largely apply for similar distributions, also.

At the time of this writing, CentOS 7, is the latest release of the distribution and was released in 07/2014. It will receive full updates until Q4/2020 and stay maintained (provided with Critical impact Security Advisories (RHSAs) and selected Urgent Priority Bug Fix Advisories (RHBAs) only) until 07/2024.
CentOS 6 also is still around; released in 07/2011, it’s maintenance will be continued until 11/2020.
CentOS 5 (released in 04/2007) exceeded it’s maintenance timeframe in 03/2017 and is considered unsupported.
Thus, I will only consider CentOS 7 and 6 here.
For details, please see CentOS Product Specifications and Red Hat Enterprise Linux Life Cycle.

Talking about Python …

Python logo

System Default and base repo

CentOS utilizes Python a lot for its command line tools. In fact, it’s primary package manager yum is heavily depending on Python 2 (2.7.5 in CentOS 7, 2.6.6 in CentOS 6 – provided by the package named “python” from “base” repo) and uninstalling it by force is a reliable way to render your package management useless. This version of Python also comes with some modules installed, which are hard to find in the most common locations like PyPI, including:

  • yum-metadata-parser
  • slip.dbus

In short: You do not want to mess around with this system interpreter! 💣

What about Python 3?

Python 3 is not available in either CentOS 6 or 7.

Python 3.0 was released in 12/2008.
CentOS 6 was released in 07/2011.
CentOS 7 was released in 07/2014.

Normally one would consider that to be a fair amount of time to add any release of a major technique to even an Enterprise Linux; especially with the Python 2 End of Life in sight for 2020. But – as you can see, it hasn’t been done yet. Remember what I said about the downsides of an Enterprise Linux? Here’s an example 😉
But you still have choices.

Using EPEL

The most convenient way to get Python 3 for any supported Enterprise Linux is by adding the EPEL repository to your system. This also has the benefit that its usage is quite common and so the risk to end up with a too customized system is not that huge.
Also, EPEL’s Guidelines and Policies aim to not interfere with any base package and also have quite a strict upgrade policy.
Also, the project is somewhat very close to RedHats own development, since it was born out of Fedora and Fedora is sponsored by RedHat and it is aimed to be used in their Enterprise Distribution, also.

EPEL provides Python 3.4.5 for both, CentOS 6 and 7.

If that satisfies your needs, this is quite a low hanging fruit. You add EPEL and install Python 3 from it like this:

Using IUS

IUS (Inline with Upstream Stable) is a project which is sponsored by Rackspace. It aims at providing more recent versions of some major key software packages, including Python.

It’s goals and philosophies are very close to those of EPEL in not to interfere with base packages. It’s naming convention makes sure that even if a package equivalent will ever show up in base, it will never interfere with those provided by IUS.
It considers itself a “SafeRepo” and compares itself with EPEL here; feel free to read this resource if want to learn more.

IUS provides several versions of Python:

  • 3.4.7 (package python34u)
  • 3.5.4 (package python35u)
  • 3.6.4 (package python36u)

If this satisfies your needs, this is also quite easy to achieve by issuing the following:

Installing from source

This is by far the most flexible approach while also the most cumbersome one. Also, this is not an “Enterprise Linux” specific task but can be done on any Linux system in the more or like the same way.
Please consider the fact that this is some kind of an anti-pattern if you think of what’s the goal and philosophy behind Enterprise Linux distributions.

By compiling from source, you can freely decide which version of Python you want to use. But at the same time, you sacrifice all benefits package managers have to offer, including:

  • Ease of installation
  • Package QA and reviewing workflow
  • Being supplied with security updates
  • Integration and availability to the package manager, often integrated into other orchestration tools

This approach will not be described here since it has already been done an uncountable amount of times (see here, or here, or any of the other ~973.000 results from a Google search) and since it is not really an Enterprise Linux approach.

Hello World!

Time to start a new Blog with a “Hello World!” post!

Let’s start with some technical, personal and historical background about my IT journey so far (even no one will be interested enough in this to read the whole article, though. And you know why I do it anyway? Because I decide so, it’s only about me to decide what is put here and what isn’t and there’s nothing you can do about 😛).
You digital-native-social-network-scum can search for some “Dislike 👎 or Report 📢” buttons as long as you like – this page was made by elders for elders 🧓 (quoth the 35 years old author), who went through the dark ages of the Internet, including connection breakdowns, because some other member of the household picked up the phone and interrupted your dial-up connection.

My first steps into the IT

I started to explore computers in 1994/1995  Continue reading “Hello World!”