Programming

Understanding Python's asyncio

Reuven M. Lerner — Wed, 07 Aug 2019 22:00:00 +0000

How to get started using Python's asyncio.

Earlier this year, I attended PyCon, the international Python conference. One topic, presented at numerous talks and discussed informally in the hallway, was the state of threading in Python—which is, in a nutshell, neither ideal nor as terrible as some critics would argue.

A related topic that came up repeatedly was that of "asyncio", a relatively new approach to concurrency in Python. Not only were there formal presentations and informal discussions about asyncio, but a number of people also asked me about courses on the subject.

I must admit, I was a bit surprised by all the interest. After all, asyncio isn't a new addition to Python; it's been around for a few years. And, it doesn't solve all of the problems associated with threads. Plus, it can be confusing for many people to get started with it.

And yet, there's no denying that after a number of years when people ignored asyncio, it's starting to gain steam. I'm sure part of the reason is that asyncio has matured and improved over time, thanks in no small part to much dedicated work by countless developers. But, it's also because asyncio is an increasingly good and useful choice for certain types of tasks—particularly tasks that work across networks.

So with this article, I'm kicking off a series on asyncio—what it is, how to use it, where it's appropriate, and how you can and should (and also can't and shouldn't) incorporate it into your own work.

What Is asyncio?

Everyone's grown used to computers being able to do more than one thing at a time—well, sort of. Although it might seem as though computers are doing more than one thing at a time, they're actually switching, very quickly, across different tasks. For example, when you ssh in to a Linux server, it might seem as though it's only executing your commands. But in actuality, you're getting a small "time slice" from the CPU, with the rest going to other tasks on the computer, such as the systems that handle networking, security and various protocols. Indeed, if you're using SSH to connect to such a server, some of those time slices are being used by sshd to handle your connection and even allow you to issue commands.

All of this is done, on modern operating systems, via "pre-emptive multitasking". In other words, running programs aren't given a choice of when they will give up control of the CPU. Rather, they're forced to give up control and then resume a little while later. Each process running on a computer is handled this way. Each process can, in turn, use threads, sub-processes that subdivide the time slice given to their parent process.

Go to Full Article

Python's Mypy: Callables and Generators

Reuven M. Lerner — Mon, 29 Jul 2019 11:00:00 +0000

by Reuven M. Lerner

Learn how Mypy's type checking works with functions and generators.

In my last two articles I've described some of the ways Mypy, a type checker for Python, can help identify potential problems with your code. [See "Introducing Mypy, an Experimental Optional Static Type Checker for Python" and "Python's Mypy—Advanced Usage".] For people (like me) who have enjoyed dynamic languages for a long time, Mypy might seem like a step backward. But given the many mission-critical projects being written in Python, often by large teams with limited communication and Python experience, some kind of type checking is an increasingly necessary evil.

It's important to remember that Python, the language, isn't changing, and it isn't becoming statically typed. Mypy is a separate program, running outside Python, typically as part of a continuous integration (CI) system or invoked as part of a Git commit hook. The idea is that Mypy runs before you put your code into production, identifying where the data doesn't match the annotations you've made to your variables and function parameters.

I'm going to focus on a few of Mypy's advanced features here. You might not encounter them very often, but even if you don't, it'll give you a better picture of the complexities associated with type checking, and how deeply the Mypy team is thinking about their work, and what tests need to be done. It'll also help you understand more about the ways people do type checking, and how to balance the beauty, flexibility and expressiveness of dynamic typing with the strictness and fewer errors of static typing.

Callable Types

When I tell participants in my Python classes that everything in Python is an object, they nod their heads, clearly thinking, "I've heard this before about other languages." But then I show them that functions and classes are both objects, and they realize that Python's notion of "everything" is a bit more expansive than theirs. (And yes, Python's definition of "everything" isn't as wide as Smalltalk's.)

When you define a function, you're creating a new object, one of type "function":


>>> def foo():
...     return "I'm foo!"

>>> type(foo)

Similarly, when you create a new class, you're adding a new object type to Python:


>>> class Foo():
...     pass

>>> type(Foo)

It's a pretty common paradigm in Python to write a function that, when it runs, defines and runs an inner function. This is also known as a "closure", and it has a few different uses. For example, you can write:

Go to Full Article

Finishing Up the Bash Mail Merge Script

Dave Taylor — Thu, 04 Jul 2019 12:00:00 +0000

by Dave Taylor

Finally, I'm going to finish the mail merge script, just in time for Replicant Day.

Remember the mail merge script I started writing a while back? Yeah, that was quite some time ago. I got sidetracked with the Linux Journal Anniversary special issue (see my article "Back in the Day: UNIX, Minix and Linux"), and then I spun off on a completely different tangent for my last article ("Breaking Up Apache Log Files for Analysis"). I blame it on...

SQUIRREL!

Oh, sorry, back to topic here. I was developing a shell script that would let you specify a text document with embedded field names that could be substituted iteratively across a file containing lots of field values.

Each field was denoted by #fieldname#, and I identified two categories of fieldnames: fixed and dynamic. A fixed value might be #name#, which would come directly out of the data file, while a dynamic value could be #date#, which would be the current date.

More interesting, I also proposed calculated values, specifically #suggested#, which would be a value calculated based on #donation#, and #date#, which would be replaced by the current date. The super-fancy version would have a simple language where you could define the relationship between variables, but let's get real. Mail merge. It's just mail merge.

Reading and Assigning Values

It turns out that the additions needed for this script aren't too difficult. The basic data file has comma-separated field names, then subsequent lines have the values associated with those fields.

Here's that core code:


if [ $lines -eq 1 ] ; then # field names
# grab variable names
declare -a varname=($f1 $f2 $f3 $f4 $f5 $f6 $f7)
else # process fields

# grab values for this line (can contain spaces)
declare -a value=("$f1" "$f2" "$f3" "$f4" "$f5" "$f6" "$f7")

The declare function turns out to be ideal for this, allowing you to create an array varname based on the contents of the first line, then keep replacing the values of the array value, so that varname[1] = value[1], and so on.

To add the additional variables #date# and #suggested#, you simply can append them to the varname and value arrays. The first one is easy, but it did highlight a weakness in the original code that I had to fix by adding quotes as shown:

Go to Full Article

Python's Mypy--Advanced Usage

Reuven M. Lerner — Mon, 24 Jun 2019 11:30:00 +0000

by Reuven M. Lerner

Mypy can check more than simple Python types.

In my last article, I introduced Mypy, a package that enforces type checking in Python programs. Python itself is, and always will remain, a dynamically typed language. However, Python 3 supports "annotations", a feature that allows you to attach an object to variables, function parameters and function return values. These annotations are ignored by Python itself, but they can be used by external tools.

Mypy is one such tool, and it's an increasingly popular one. The idea is that you run Mypy on your code before running it. Mypy looks at your code and makes sure that your annotations correspond with actual usage. In that sense, it's far stricter than Python itself, but that's the whole point.

In my last article, I covered some basic uses for Mypy. Here, I want to expand upon those basics and show how Mypy really digs deeply into type definitions, allowing you to describe your code in a way that lets you be more confident of its stability.

Type Inference

Consider the following code:


x: int = 5
x = 'abc'
print(x)

This first defines the variable x, giving it a type annotation of int. It also assigns it to the integer 5. On the next line, it assigns x the string abc. And on the third line, it prints the value of x.

The Python language itself has no problems with the above code. But if you run mypy against it, you'll get an error message:


mytest.py:5: error: Incompatible types in assignment
   (expression has type "str", variable has type "int")

As the message says, the code declared the variable to have type int, but then assigned a string to it. Mypy can figure this out because, despite what many people believe, Python is a strongly typed language. That is, every object has one clearly defined type. Mypy notices this and then warns that the code is assigning values that are contrary to what the declarations said.

In the above code, you can see that I declared x to be of type int at definition time, but then assigned it to a string, and then I got an error. What if I don't add the annotation at all? That is, what if I run the following code via Mypy:

Go to Full Article

Build Your Own Internet Radio Receiver

Nick Tufillaro — Mon, 27 May 2019 11:00:00 +0000

by Nick Tufillaro

Tune in to communities around the world with the push of a button.

When I get home at night, I like to tune into the world with the push of a button. I've lived in lots of different places—from Dunedin, New Zealand, to Santa Fe, New Mexico—and in each town, I've come to love a radio station (usually a community radio station) that embodies the spirit of the place. With the push of a button, I can get a bit back in sync with each of these places and also visit new communities, thanks to internet radio.

Why build your own internet radio receiver? One option, of course, is simply to use an app for a receiver. However, I've found that the most common apps don't keep their focus on the task at hand, and are increasingly distracted by offering additional social-networking services. And besides, I want to listen now. I don't want to check into my computer or phone, log in yet again, and endure the stress of recalling YAPW (Yet Another PassWord). I've also found that the current offering of internet radio boxes falls short of my expectations. Like I said, I've lived in a lot of places—more than two or four or eight. I want a lot of buttons, so I can tune in to a radio station with just one gesture. Finally, I've noticed that streams are increasingly problematic if I don't go directly to the source. Often, streams chosen through a "middle man" start with an ad or blurb that is tacked on as a preamble. Or sometimes the "middle man" might tie me to a stream of lower audio quality than the best being served up.

So, I turned to building my own internet radio receiver—one with lots of buttons that allow me to "tune in" without being too pushy. In this article, I share my experience. In principle, it should be easy—you just need a Linux distro, a ship to sail her on and an external key pad for a rudder. In practice, it's not too hard, but there are a few obstacles along the course that I hope to help you navigate.

My recipe list included the following:

A used notebook with an ultra low voltage (Core 2 Duo) processor.
An audio interface with an optical TOSLINK.
pyradio: an open-source Python radio program.
An external keypad.

Figure 1. My Hardware Setup

Why a notebook and not a Raspberry Pi or ship of a similar ilk? Mostly due to time—my time in particular. It's not too hard to find a high quality notebook about ten years old for about $50, so the cost is really not that different, and I find the development platform to be much quicker.

Go to Full Article

Introducing Mypy, an Experimental Optional Static Type Checker for Python

Reuven M. Lerner — Mon, 13 May 2019 11:30:00 +0000

by Reuven M. Lerner

Tighten up your code and identify errors before they occur with mypy.

I've been using dynamic languages—Perl, Ruby and Python—for many years. I love the flexibility and expressiveness that such languages provide. For example, I can define a function that sums numbers:


def mysum(numbers):
    total = 0
   for one_number in numbers:
       total += one_number
   return total

The above function will work on any iterable that returns numbers. So I can run the above on a list, tuple or set of numbers. I can even run it on a dictionary whose keys are all numbers. Pretty great, right?

Yes, but for my students who are used to static, compiled languages, this is a very hard thing to get used to. After all, how can you make sure that no one passes you a string, or a number of strings? What if you get a list in which some, but not all, of the elements are numeric?

For a number of years, I used to dismiss such worries. After all, dynamic languages have been around for a long time, and they have done a good job. And really, if people are having these sorts of type mismatch errors, then maybe they should be paying closer attention. Plus, if you have enough testing, you'll probably be fine.

But as Python (and other dynamic languages) have been making inroads into large companies, I've become increasingly convinced that there's something to be said for type checking. In particular, the fact that many newcomers to Python are working on large projects, in which many parts need to interoperate, has made it clear to me that some sort of type checking can be useful.

How can you balance these needs? That is, how can you enjoy Python as a dynamically typed language, while simultaneously getting some added sense of static-typing stability?

One of the most popular answers is a system known as mypy, which takes advantage of Python 3's type annotations for its own purposes. Using mypy means that you can write and run Python in the normal way, gradually adding static type checking over time and checking it outside your program's execution.

In this article, I start exploring mypy and how you can use it to check for problems in your programs. I've been impressed by mypy, and I believe you're likely to see it deployed in a growing number of places, in no small part because it's optional, and thus allows developers to use it to whatever degree they deem necessary, tightening things up over time, as well.

Dynamic and Strong Typing

In Python, users enjoy not only dynamic typing, but also strong typing. "Dynamic" means that variables don't have types, but that values do. So you can say:

Go to Full Article

What The @#$%&! (Heck) is this #! (Hash-Bang) Thingy In My Bash Script

Mitch Frazier — Fri, 10 May 2019 11:30:00 +0000

by Mitch Frazier

You've seen it a million times—the hash-bang (#!) line at the top of a script—whether it be Bash, Python, Perl or some other scripting language. And, I'm sure you know what its purpose is: it specifies the script interpreter that's used to execute the script. But, do you know how it actually works? Your initial thought might be that your shell (bash) reads that line and then executes the specified interpreter, but that's not at all how it works. How it actually works is the main focus of this post, but I also want to introduce how you can create your own version of "hash-bang" if you're so inclined.

Go to Full Article

Linux...Do It for the Children

Marcel Gagné — Thu, 11 Apr 2019 11:30:00 +0000

by Marcel Gagné

A rundown of some fun and educational Linux software for kids.

I'm probably going to regret that title. I've been making fun of those words, "do it for the children" for years. It's one of those "reasons" people turn to when all else has failed in terms of getting you to sign on to whatever lifestyle, agenda, law, changes to food—you name it. Hearing those words draws the Spock eyebrow lift out of me faster than you can say, "fascinating".

Okay, pretend that I didn't start this article with that comment. Let's try this instead.

As I write this, my youngest son is 11 years old. He has grown up in a magical world of electronics that delivers what he wants to watch when he wants to watch it. Access to the web is something he always has known. Until very recently, he never had seen television with commercials. A couple years ago, my wife and I thought it was something he should at least understand, so we turned to a live TV program for the first time in I don't know how long. He was not impressed with the interruptions. Now, with multiple Google Home units in the house, including one in his bedroom, the on-demand magic is pretty much complete.

He started playing video games when he was three and was scary good on my PS3 by the time he turned four. He started using a laptop when he was five, and that laptop ran Linux. I'm pretty sure he was using Kubuntu, but it might have been Linux Mint. Either way, it was a KDE Plasma desktop. In short, the world of tech is nothing new for him, and Linux is just what people run. His school has Chromebooks, and the few run-ins he's had with Windows left him cold.

Kids and Linux? Absolutely.

GCompris

Much earlier on, however, I took advantage of some of the simpler educational games available on Linux. One of my favorites is GCompris, an all-in-one collection of educational games for children, geared for ages two to ten (Figure 1). By the way, GCompris is pronounced like the French words, J'ai compris, and it literally means, "I have understood", paying homage to its educational focus. I've mentioned this one in the past, but GCompris is a living, breathing project, actively developed by the KDE community with a new release just this past month.

Figure 1. GCompris is a suite of educational games for kids.

Go to Full Article

The Kids Take Over

Doc Searls — Mon, 08 Apr 2019 11:30:00 +0000

by Doc Searls

As with Linux, these kids are all about making things—and then making them better. They're also up against incumbent top-down systems they will reform or defeat. Those are the only choices.

It starts here, in the heart of Long Island, a couple dozen exits east of Queens. I saw it with my own eyes in Mineola's Public Schools, where kids, led by a nonprofit called kidOYO ("kid-oh-yo"), are learning to program in different languages on different devices and operating systems, creating and re-creating software and hardware, with fun and at speed. Their esteem in themselves and in the eyes of their peers derives from their actual work and their helpfulness to others. They are also moving ahead through levels of productivity and confidence that are sure to create real-world results and strip the gears of any system meant to contain them. Mineola's schools are not one of those systems.

OYO means Own Your Own, and that's what these kids are learning to do. In geekier terms, they are rooting their own lives online. They're doing it by learning to program in languages that start with Scratch and progress through Python, Java, C# and beyond. They're doing it on every hardware and software platform they can, while staying anchored to Linux, because Linux is where the roots of personal freedom and agency go deepest. And they're doing in all in the spirit of Linus' book title: just for fun.

With kidOYO, the heuristics go both ways: kidOYO teaches the kids, and the kids teach kidOYO. Iteration is constant. What works gets improved, and what doesn't gets tossed. The measures of success are how enthused the kids stay, how much they give and get energy from each other, and how much they learn and teach. Nowhere are they sorted into bell curves or given caste-producing labels, such as "gifted" or "challenged". Nor are they captive to the old report-card system. When they do take standardized tests, for example the college AP (advanced placement) ones for computer science, they tend to kick ass.

kidOYO is the creation of the Loffreto family: Devon and Melora, and their son Zhen, who is now 13. What started as a way to teach computing to Zhen turned into ways to teach computer science to every kid, everywhere. kidOYO's methods resemble how the Linux kernel constantly improves, with code contributors and maintainers stamping out bugs and iterating toward ever-expanding completeness, guided by an equal mix of purpose and fun.

Go to Full Article

Fun with Mail Merge and Cool Bash Arrays

Dave Taylor — Mon, 25 Mar 2019 11:30:00 +0000

by Dave Taylor

Creating a sed-based file substitution tool.

A few weeks ago, I was digging through my spam folder and found an email message that started out like this:


Dear #name#
Congratulations on winning the $15.7 million lottery payout!
To learn how to claim your winnings, please...

Obviously, it was a scam (does anyone actually fall for these?), but what captured my attention was the #name# sequence. Clearly that was a fail on the part of the sender who presumably didn't know how to use AnnoyingSpamTool 1.3 or whatever the heck he or she was using.

The more general notation for bulk email and file transformations is pretty interesting, however. There are plenty of legitimate reasons to use this sort of substitution, ranging from email newsletters (like the one I send every week from AskDaveTaylor.com—check it out!) to stockholder announcements and much more.

With that as the inspiration, let's build a tool that offers just this capability.

The simple version will be a 1:1 substitution, so #name# becomes, say, "Rick Deckard", while #first# might be "Rick" and #last# might be "Deckard". Let's build on that, but let's start small.

Simple Word Substitution in Linux

There are plenty of ways to tackle the word substitution from the command line, ranging from Perl to awk, but here I'm using the original UNIX command sed (stream editor) designed for exactly this purpose. General notation for a substitution is s/old/new/, and if you tack on a g at the end, it matches every occurrence on a line, not only the first, so the full command is s/old/new/g.

Before going further, here's a simple document that has necessary substitutions embedded:


$ cat convertme.txt
#date#

Dear #name#, I wanted to start by again thanking you for your
generous donation of #amount# in #month#. We couldn't do our
work without support from humans like you, #first#.

This year we're looking at some unexpected expenses,
particularly in Sector 5, which encompasses #state#, as you
know. I'm hoping you can start the year with an additional
contribution? Even #suggested# would be tremendously helpful.

Thanks for your ongoing support. With regards,

Rick Deckard
Society for the Prevention of Cruelty to Replicants

Scan through it, and you'll see there's a lot of substitutions to do: #date#, #name#, #amount#, #month#, #first#, #state# and #suggested#. It turns out that #date# will be replaced with the current date, and #suggested# is one that'll be calculated as the letter is processed, but that's for a bit later, so stay tuned for that.

Go to Full Article