python

Experimenting with Python implementation of Host Identity Protocol

Dmitriy Kuptsov — Thu, 26 Aug 2021 16:00:00 +0000

Host Identity Protocol, or HIP, is a layer 3.5 solution and was initially designed to split the dual role of the IP address - locator and identifier. Using HIP protocol one can solve not only mobility problems but also establish authenticated secure channel between two communicating end-hosts. In this short article, we first introduce relevant background information. We then present some mathematical background related to Elliptic Curve (EC) Cryptography and Diffie-Hellman protocol based on EC. Finally, we demonstrate some micro benchmarking results for various cryptographic primitives and conclude the article with the results for the overall performance of HIP and IPSec, which we implemented in Linux userspace using Python language.

INTRODUCTION

Sometimes it is easier to implement prototypes in userspace using high-level languages, such as Python or Java. In this document we attempt to describe our implementation effort related to Host Identity Protocol version 2. In the first part, we describe various security solutions, then we discuss some implementation details of the HIP protocol, and finally, in the last part of this work we discuss the performance of the HIP and IPSec protocols implemented using Python language.

BACKGROUND

In this section we will describe basic background. First, we will discuss the problem of mobile Internet and introduce the Host Identity Protocol. We then move to the discussion of various security protocols. We will conclude the section with the discussion of Elliptic Curves and a variant of Diffie-Hellman algorithm, which uses EC cryptography (ECC).

Internet was designed initially so that the Internet Protocol (IP) address is playing dual role: it is the locator, so that the routers can find the recipient of a message, and it is an identifier, so that the upper layer protocols (such as TCP and UDP) can make bindings (for example, transport layer sockets use IP addresses and ports to make a connections). This becomes a problem when a networked device roams from one network to another, and so the IP address changes, leading to failures in upper layer connections. The other problem is establishment of the authenticated channel between the communicating parties. In practice, when making connections, long term identities of the parties are not verified. Of course, there are solutions such as SSL which can readily solve the problem at hand. However, SSL is suitable only for TCP connections and most of the time practical use cases include only secure web surfing and establishment of VPN tunnels. Host Identity Protocol on the other hand is more flexible: it allows peers to create authenticated secure channel on network layer, and so all upper layer protocols can benefit from such channel.

Go to Full Article

Understanding Python's asyncio

Reuven M. Lerner — Wed, 07 Aug 2019 22:00:00 +0000

by Reuven M. Lerner

How to get started using Python's asyncio.

Earlier this year, I attended PyCon, the international Python conference. One topic, presented at numerous talks and discussed informally in the hallway, was the state of threading in Python—which is, in a nutshell, neither ideal nor as terrible as some critics would argue.

A related topic that came up repeatedly was that of "asyncio", a relatively new approach to concurrency in Python. Not only were there formal presentations and informal discussions about asyncio, but a number of people also asked me about courses on the subject.

I must admit, I was a bit surprised by all the interest. After all, asyncio isn't a new addition to Python; it's been around for a few years. And, it doesn't solve all of the problems associated with threads. Plus, it can be confusing for many people to get started with it.

And yet, there's no denying that after a number of years when people ignored asyncio, it's starting to gain steam. I'm sure part of the reason is that asyncio has matured and improved over time, thanks in no small part to much dedicated work by countless developers. But, it's also because asyncio is an increasingly good and useful choice for certain types of tasks—particularly tasks that work across networks.

So with this article, I'm kicking off a series on asyncio—what it is, how to use it, where it's appropriate, and how you can and should (and also can't and shouldn't) incorporate it into your own work.

What Is asyncio?

Everyone's grown used to computers being able to do more than one thing at a time—well, sort of. Although it might seem as though computers are doing more than one thing at a time, they're actually switching, very quickly, across different tasks. For example, when you ssh in to a Linux server, it might seem as though it's only executing your commands. But in actuality, you're getting a small "time slice" from the CPU, with the rest going to other tasks on the computer, such as the systems that handle networking, security and various protocols. Indeed, if you're using SSH to connect to such a server, some of those time slices are being used by sshd to handle your connection and even allow you to issue commands.

All of this is done, on modern operating systems, via "pre-emptive multitasking". In other words, running programs aren't given a choice of when they will give up control of the CPU. Rather, they're forced to give up control and then resume a little while later. Each process running on a computer is handled this way. Each process can, in turn, use threads, sub-processes that subdivide the time slice given to their parent process.

Go to Full Article

DNA Geometry with cadnano

Joey Bernard — Wed, 07 Aug 2019 19:30:00 +0000

by Joey Bernard

This article introduces a tool you can use to work on three-dimensional DNA origami. The package is called cadnano, and it's currently being developed at the Wyss Institute. With this package, you'll be able to construct and manipulate the three-dimensional representations of DNA structures, as well as generate publication-quality graphics of your work.

Because this software is research-based, you won't likely find it in the package repository for your favourite distribution, in which case you'll need to install it from the GitHub repository.

Since cadnano is a Python program, written to use the Qt framework, you'll need to install some packages first. For example, in Debian-based distributions, you'll want to run the following commands:


sudo apt-get install python3 python3-pip

I found that installation was a bit tricky, so I created a virtual Python environment to manage module installations.

Once you're in your activated virtualenv, install the required Python modules with the command:


pip3 install pythreejs termcolor pytz pandas pyqt5 sip

After those dependencies are installed, grab the source code with the command:


git clone https://github.com/cadnano/cadnano2.5.git

This will grab the Qt5 version. The Qt4 version is in the repository https://github.com/cadnano/cadnano2.git.

Changing directory into the source directory, you can build and install cadnano with:


python setup.py install

Now your cadnano should be available within the virtualenv.

You can start cadnano simply by executing the cadnano command from a terminal window. You'll see an essentially blank workspace, made up of several empty view panes and an empty inspector pane on the far right-hand side.

Figure 1. When you first start cadnano, you get a completely blank work space.

In order to walk through a few of the functions available in cadnano, let's create a six-strand nanotube. The first step is to create a background that you can use to build upon. At the top of the main window, you'll find three buttons in the toolbar that will let you create a "Freeform", "Honeycomb" or "Square" framework. For this example, click the honeycomb button.

Figure 2. Start your construction with one of the available geometric frameworks.

Go to Full Article

Python's Mypy: Callables and Generators

Reuven M. Lerner — Mon, 29 Jul 2019 11:00:00 +0000

by Reuven M. Lerner

Learn how Mypy's type checking works with functions and generators.

In my last two articles I've described some of the ways Mypy, a type checker for Python, can help identify potential problems with your code. [See "Introducing Mypy, an Experimental Optional Static Type Checker for Python" and "Python's Mypy—Advanced Usage".] For people (like me) who have enjoyed dynamic languages for a long time, Mypy might seem like a step backward. But given the many mission-critical projects being written in Python, often by large teams with limited communication and Python experience, some kind of type checking is an increasingly necessary evil.

It's important to remember that Python, the language, isn't changing, and it isn't becoming statically typed. Mypy is a separate program, running outside Python, typically as part of a continuous integration (CI) system or invoked as part of a Git commit hook. The idea is that Mypy runs before you put your code into production, identifying where the data doesn't match the annotations you've made to your variables and function parameters.

I'm going to focus on a few of Mypy's advanced features here. You might not encounter them very often, but even if you don't, it'll give you a better picture of the complexities associated with type checking, and how deeply the Mypy team is thinking about their work, and what tests need to be done. It'll also help you understand more about the ways people do type checking, and how to balance the beauty, flexibility and expressiveness of dynamic typing with the strictness and fewer errors of static typing.

Callable Types

When I tell participants in my Python classes that everything in Python is an object, they nod their heads, clearly thinking, "I've heard this before about other languages." But then I show them that functions and classes are both objects, and they realize that Python's notion of "everything" is a bit more expansive than theirs. (And yes, Python's definition of "everything" isn't as wide as Smalltalk's.)

When you define a function, you're creating a new object, one of type "function":


>>> def foo():
...     return "I'm foo!"

>>> type(foo)

Similarly, when you create a new class, you're adding a new object type to Python:


>>> class Foo():
...     pass

>>> type(Foo)

It's a pretty common paradigm in Python to write a function that, when it runs, defines and runs an inner function. This is also known as a "closure", and it has a few different uses. For example, you can write:

Go to Full Article

An AI Wizard of Words

Marcel Gagné — Mon, 15 Jul 2019 11:00:00 +0000

by Marcel Gagné

A look at using OpenAI's Generative Pretrained Transformer 2 (GPT-2) to generate text.

It's probably fair to say that there's more than one person out there who is worried about some version of artificial intelligence, or AI, possibly in a robot body of some kind, taking people's jobs. Anything that is repetitive or easily described is considered fair game for a robot, so driving a car or working in a factory is fair game.

Until recently, we could tell ourselves that people like yours truly—the writers and those who create things using some form of creativity—were more or less immune to the march of the machines. Then came GPT-2, which stands for Generative Pretrained Transformer 2. I think you'll agree, that isn't the sexiest name imaginable for a civilization-ending text bot. And since it's version 2, I imagine that like Star Trek's M-5 computer, perhaps GPT-1 wasn't entirely successful. That would be the original series episode titled, "The Ultimate Computer", if you want to check it out.

So what does the name "GPT-2" stand for? Well, "generative" means pretty much what it sounds like. The program generates text based on a predictive model, much like your phone suggests the next word as you type. The "pretrained" part is also quite obvious in that the model released by OpenAI has been built and fine-tuned for a specific purpose. The last word, "Transformer", refers to the "transformer architecture", which is a neural network design architecture suited for understanding language. If you want to dig deeper into that last one, I've included a link from a Google AI blog that compares it to other machine learning architecture (see Resources).

On February 14, 2019, Valentine's Day, OpenAI released GPT-2 with a warning:

Our model, called GPT-2 (a successor to GPT), was trained simply to predict the next word in 40GB of Internet text. Due to our concerns about malicious applications of the technology, we are not releasing the trained model. As an experiment in responsible disclosure, we are instead releasing a much smaller model for researchers to experiment with, as well as a technical paper.

I've included a link to the blog in the Resources section at the end of this article. It's worth reading partly because it demonstrates a sample of what this software is capable of using the full model (see Figure 1 for a sample). We already have a problem with human-generated fake news; imagine a tireless machine capable of churning out vast quantities of news and posting it all over the internet, and you start to get a feel for the dangers. For that reason, OpenAI released a much smaller model to demonstrate its capabilities and to engage researchers and developers.

Go to Full Article

Python's Mypy--Advanced Usage

Reuven M. Lerner — Mon, 24 Jun 2019 11:30:00 +0000

by Reuven M. Lerner

Mypy can check more than simple Python types.

In my last article, I introduced Mypy, a package that enforces type checking in Python programs. Python itself is, and always will remain, a dynamically typed language. However, Python 3 supports "annotations", a feature that allows you to attach an object to variables, function parameters and function return values. These annotations are ignored by Python itself, but they can be used by external tools.

Mypy is one such tool, and it's an increasingly popular one. The idea is that you run Mypy on your code before running it. Mypy looks at your code and makes sure that your annotations correspond with actual usage. In that sense, it's far stricter than Python itself, but that's the whole point.

In my last article, I covered some basic uses for Mypy. Here, I want to expand upon those basics and show how Mypy really digs deeply into type definitions, allowing you to describe your code in a way that lets you be more confident of its stability.

Type Inference

Consider the following code:


x: int = 5
x = 'abc'
print(x)

This first defines the variable x, giving it a type annotation of int. It also assigns it to the integer 5. On the next line, it assigns x the string abc. And on the third line, it prints the value of x.

The Python language itself has no problems with the above code. But if you run mypy against it, you'll get an error message:


mytest.py:5: error: Incompatible types in assignment
   (expression has type "str", variable has type "int")

As the message says, the code declared the variable to have type int, but then assigned a string to it. Mypy can figure this out because, despite what many people believe, Python is a strongly typed language. That is, every object has one clearly defined type. Mypy notices this and then warns that the code is assigning values that are contrary to what the declarations said.

In the above code, you can see that I declared x to be of type int at definition time, but then assigned it to a string, and then I got an error. What if I don't add the annotation at all? That is, what if I run the following code via Mypy:

Go to Full Article

Build Your Own Internet Radio Receiver

Nick Tufillaro — Mon, 27 May 2019 11:00:00 +0000

by Nick Tufillaro

Tune in to communities around the world with the push of a button.

When I get home at night, I like to tune into the world with the push of a button. I've lived in lots of different places—from Dunedin, New Zealand, to Santa Fe, New Mexico—and in each town, I've come to love a radio station (usually a community radio station) that embodies the spirit of the place. With the push of a button, I can get a bit back in sync with each of these places and also visit new communities, thanks to internet radio.

Why build your own internet radio receiver? One option, of course, is simply to use an app for a receiver. However, I've found that the most common apps don't keep their focus on the task at hand, and are increasingly distracted by offering additional social-networking services. And besides, I want to listen now. I don't want to check into my computer or phone, log in yet again, and endure the stress of recalling YAPW (Yet Another PassWord). I've also found that the current offering of internet radio boxes falls short of my expectations. Like I said, I've lived in a lot of places—more than two or four or eight. I want a lot of buttons, so I can tune in to a radio station with just one gesture. Finally, I've noticed that streams are increasingly problematic if I don't go directly to the source. Often, streams chosen through a "middle man" start with an ad or blurb that is tacked on as a preamble. Or sometimes the "middle man" might tie me to a stream of lower audio quality than the best being served up.

So, I turned to building my own internet radio receiver—one with lots of buttons that allow me to "tune in" without being too pushy. In this article, I share my experience. In principle, it should be easy—you just need a Linux distro, a ship to sail her on and an external key pad for a rudder. In practice, it's not too hard, but there are a few obstacles along the course that I hope to help you navigate.

My recipe list included the following:

A used notebook with an ultra low voltage (Core 2 Duo) processor.
An audio interface with an optical TOSLINK.
pyradio: an open-source Python radio program.
An external keypad.

Figure 1. My Hardware Setup

Why a notebook and not a Raspberry Pi or ship of a similar ilk? Mostly due to time—my time in particular. It's not too hard to find a high quality notebook about ten years old for about $50, so the cost is really not that different, and I find the development platform to be much quicker.

Go to Full Article

Introducing Mypy, an Experimental Optional Static Type Checker for Python

Reuven M. Lerner — Mon, 13 May 2019 11:30:00 +0000

by Reuven M. Lerner

Tighten up your code and identify errors before they occur with mypy.

I've been using dynamic languages—Perl, Ruby and Python—for many years. I love the flexibility and expressiveness that such languages provide. For example, I can define a function that sums numbers:


def mysum(numbers):
    total = 0
   for one_number in numbers:
       total += one_number
   return total

The above function will work on any iterable that returns numbers. So I can run the above on a list, tuple or set of numbers. I can even run it on a dictionary whose keys are all numbers. Pretty great, right?

Yes, but for my students who are used to static, compiled languages, this is a very hard thing to get used to. After all, how can you make sure that no one passes you a string, or a number of strings? What if you get a list in which some, but not all, of the elements are numeric?

For a number of years, I used to dismiss such worries. After all, dynamic languages have been around for a long time, and they have done a good job. And really, if people are having these sorts of type mismatch errors, then maybe they should be paying closer attention. Plus, if you have enough testing, you'll probably be fine.

But as Python (and other dynamic languages) have been making inroads into large companies, I've become increasingly convinced that there's something to be said for type checking. In particular, the fact that many newcomers to Python are working on large projects, in which many parts need to interoperate, has made it clear to me that some sort of type checking can be useful.

How can you balance these needs? That is, how can you enjoy Python as a dynamically typed language, while simultaneously getting some added sense of static-typing stability?

One of the most popular answers is a system known as mypy, which takes advantage of Python 3's type annotations for its own purposes. Using mypy means that you can write and run Python in the normal way, gradually adding static type checking over time and checking it outside your program's execution.

In this article, I start exploring mypy and how you can use it to check for problems in your programs. I've been impressed by mypy, and I believe you're likely to see it deployed in a growing number of places, in no small part because it's optional, and thus allows developers to use it to whatever degree they deem necessary, tightening things up over time, as well.

Dynamic and Strong Typing

In Python, users enjoy not only dynamic typing, but also strong typing. "Dynamic" means that variables don't have types, but that values do. So you can say:

Go to Full Article

Become Queen Bee for a Day Using Python's Built-in Data Types

Reuven M. Lerner — Mon, 11 Mar 2019 11:30:00 +0000

by Reuven M. Lerner

Cheaters never win, but at least they can use Python.

Like many other nerds, I love word puzzles. I'm not always great at them, and I don't always have time to do them, but when I do, I really enjoy them.

I recently discovered a new daily puzzle, known as "spelling bee", that the New York Times offers online. The idea is simple. There are seven different letters, one in the center of a circle and six around it. Your job is to make as many different words as you can from those seven letters. Each word must be at least four letters long, and each word also must contain the center letter. You can use each letter as many times as you want.

So if the letters are "eoncylt", with a center letter of "y", some of the words you could create might be "cyclone", "eyelet" and "nylon".

The online game gives you a score based on how many words you've made from the potential pool. If you get them all, you're awarded "queen bee" status.

I do pretty well at this puzzle, but I've never managed to find all of the hidden words. Nevertheless, I have become queen bee on a few occasions. How? The answer is simple. I cheated. How? Using Python, of course.

Now, cheating at games isn't necessarily the first order of business when it comes to programming. And cheating at word games in which you're competing against yourself is probably a sign of unhealthy competition. But, doing so also provides a great way to review some of the ways you can use Python's built-in data types and the ease with which you can process words and text.

So in this article, I explore a number of ways you can cheat—and yes, become the queen bee, if only for a day.

Trying All Combinations

To start, you simply might try to form all of the possible combinations you can with the letters you're given. As you might remember from high-school math class, there's a difference between "permutations" and "combinations". When you generate "permutations", the order is important, but when you generate "combinations", the order is not important.

You easily can see this using Python's itertools module, a part of the standard library that has functions named permutations and combinations. Each takes both an iterable data structure and the number of items you want in each resulting list. For example:

Go to Full Article

By Jupyter--Is This the Future of Open Science?

Glyn Moody — Mon, 04 Mar 2019 12:30:00 +0000

by Glyn Moody

Taking the scientific paper to the next level.

In a recent article, I explained why open source is a vital part of open science. As I pointed out, alongside a massive failure on the part of funding bodies to make open source a key aspect of their strategies, there's also a similar lack of open-source engagement with the needs and challenges of open science. There's not much that the Free Software world can do to change the priorities of funders. But, a lot can be done on the other side of things by writing good open-source code that supports and enhances open science.

People working in science potentially can benefit from every piece of free software code—the operating systems and apps, and the tools and libraries—so the better those become, the more useful they are for scientists. But there's one open-source project in particular that already has had a significant impact on how scientists work—Project Jupyter:

Project Jupyter is a set of open-source software projects that form the building blocks for interactive and exploratory computing that is reproducible and multi-language. The main application offered by Jupyter is the Jupyter Notebook, a web-based interactive computing platform that allows users to author documents that combine live code, equations, narrative text, interactive dashboard and other rich media.

Project Jupyter was spun-off from IPython in 2014 by Fernando Pérez. Although it began as an environment for programming Python, its ambitions have grown considerably. Today, dozens of Jupyter kernels exist that allow other languages to be used. Indeed, the project itself speaks of supporting "interactive data science and scientific computing across all programming languages". As well as this broad-based support for programming languages, Jupyter is noteworthy for its power. It enables users to create and share documents that contain live code, equations, visualizations and narrative text. Uses include data cleaning and transformation, numerical simulation, statistical modeling, data visualization and machine learning.

Go to Full Article