Tuesday, 28 June 2016

Device Drivers, Part 1: Linux Device Drivers for Your Girl Friend

This series on Linux device drivers aims to present the usually technical topic in a way that is more interesting to a wider cross-section of readers.
“After a week of hard work, we finally got our driver working,” were Pugs’ first words when he met his girlfriend, Shweta.
“Why? What was your driver up to? Was he sick? And what hard work did you do?” asked Shweta. Confused, Pugs responded, “What are you talking about?”
Now it was Shweta’s turn to look puzzled, as she replied, “Why ask me? You tell me — which of your drivers are you talking about?”
When understanding dawned on him, Pugs groaned, “Ah c’mon! Not my car drivers — I am talking about a device driver on my computer.”
“I know about car and bus drivers, pilots, and even screwdrivers; but what is this ‘device driver’?” queried Shweta, puzzled.
That was all it took to launch Pugs into a passionate explanation of device drivers for the newbie — in particular, Linux device drivers, which he had been working on for many years.

Of drivers and buses

A driver drives, manages, controls, directs and monitors the entity under its command. What a bus driver does with a bus, a device driver does with a computer device (any piece of hardware connected to a computer) like a mouse, keyboard, monitor, hard disk, Web-camera, clock, and more.
Further, a “pilot” could be a person or even an automatic system monitored by a person (an auto-pilot system in airliners, for example). Similarly, a specific piece of hardware could be controlled by a piece of software (a device driver), or could be controlled by another hardware device, which in turn could be managed by a software device driver. In the latter case, such a controlling device is commonly called a device controller. This, being a device itself, often also needs a driver, which is commonly referred to as a bus driver.
General examples of device controllers include hard disk controllers, display controllers, and audio controllers that in turn manage devices connected to them. More technical examples would be an IDE controller, PCI controller, USB controller, SPI controller, I2C controller, etc. Pictorially, this whole concept can be depicted as in Figure 1.

Device and driver interaction
Figure 1: Device and driver interaction

Device controllers are typically connected to the CPU through their respectively named buses (collection of physical lines) — for example, the PCI bus, the IDE bus, etc. In today’s embedded world, we encounter more micro-controllers than CPUs; these are the CPU plus various device controllers built onto a single chip. This effective embedding of device controllers primarily reduces cost and space, making it suitable for embedded systems. In such cases, the buses are integrated into the chip itself. Does this change anything for the drivers, or more generically, on the software front?
The answer is, not much — except that the bus drivers corresponding to the embedded device controllers are now developed under the architecture-specific umbrella.

Drivers have two parts

Bus drivers provide hardware-specific interfaces for the corresponding hardware protocols, and are the bottom-most horizontal software layers of an operating system (OS). Over these sit the actual device drivers. These operate on the underlying devices using the horizontal layer interfaces, and hence are device-specific. However, the whole idea of writing these drivers is to provide an abstraction to the user, and so, at the other “end”, these do provide an interface (which varies from OS to OS). In short, a device driver has two parts, which are: a) device-specific, and b) OS-specific. Refer to Figure 2.

Linux device driver partition
Figure 2: Linux device driver partition

The device-specific portion of a device driver remains the same across all operating systems, and is more about understanding and decoding the device data sheets than software programming. A data sheet for a device is a document with technical details of the device, including its operation, performance, programming, etc. — in short a device user manual.
Later, I shall show some examples of decoding data sheets as well. However, the OS-specific portion is the one that is tightly coupled with the OS mechanisms of user interfaces, and thus differentiates a Linux device driver from a Windows device driver and from a MacOS device driver.

Verticals

In Linux, a device driver provides a “system call” interface to the user; this is the boundary line between the so-called kernel space and user-space of Linux, as shown in Figure 2. Figure 3 provides further classification.

Linux kernel overview
Figure 3: Linux kernel overview

Based on the OS-specific interface of a driver, in Linux, a driver is broadly classified into three verticals:
  • Packet-oriented or the network vertical
  • Block-oriented or the storage vertical
  • Byte-oriented or the character vertical
The CPU vertical and memory vertical, taken together with the other three verticals, give the complete overview of the Linux kernel, like any textbook definition of an OS: “An OS performs 5 management functions: CPU/process, memory, network, storage, device I/O.” Though these two verticals could be classified as device drivers, where CPU and memory are the respective devices, they are treated differently, for many reasons.
These are the core functionalities of any OS, be it a micro-kernel or a monolithic kernel. More often than not, adding code in these areas is mainly a Linux porting effort, which is typically done for a new CPU or architecture. Moreover, the code in these two verticals cannot be loaded or unloaded on the fly, unlike the other three verticals. Henceforth, when we talk about Linux device drivers, we mean to talk only about the latter three verticals in Figure 3.
Let’s get a little deeper into these three verticals. The network vertical consists of two parts: a) the network protocol stack, and b)the network interface card (NIC) device drivers, or simply network device drivers, which could be for Ethernet, Wi-Fi, or any other network horizontals. Storage, again, consists of two parts: a) File-system drivers, to decode the various formats on different partitions, and b) Block device drivers for various storage (hardware) protocols, i.e., horizontals like IDE, SCSI, MTD, etc.
With this, you may wonder if that is the only set of devices for which you need drivers (or for which Linux has drivers). Hold on a moment; you certainly need drivers for the whole lot of devices that interface with the system, and Linux does have drivers for them. However, their byte-oriented cessibility puts all of them under the character vertical — this is, in reality, the majority bucket. In fact, because of the vast number of drivers in this vertical, character drivers have been further sub-classified — so you have tty drivers, input drivers, console drivers, frame-buffer drivers, sound drivers, etc. The typical horizontals here would be RS232, PS/2, VGA, I2C, I2S, SPI, etc.

Multiple-vertical drivers

One final note on the complete picture (placement of all the drivers in the Linux driver ecosystem): the horizontals like USB, PCI, etc, span below multiple verticals. Why is that?
Simple — you already know that you can have a USB Wi-Fi dongle, a USB pen drive, and a USB-to-serial converter — all are USB, but come under three different verticals!
In Linux, bus drivers or the horizontals, are often split into two parts, or even two drivers: a) device controller-specific, and b) an abstraction layer over that for the verticals to interface, commonly called cores. A classic example would be the USB controller drivers ohci, ehci, etc., and the USB abstraction, usbcore.

Summing up

So, to conclude, a device driver is a piece of software that drives a device, though there are so many classifications. In case it drives only another piece of software, we call it just a driver. Examples are file-system drivers, usbcore, etc. Hence, all device drivers are drivers, but all drivers are not device drivers.
“Hey, Pugs, hold on; we’re getting late for class, and you know what kind of trouble we can get into. Let’s continue from here, later,” exclaimed Shweta.
Jumping up, Pugs finished his explanation: “Okay. This is the basic theory about device drivers. If you’re interested, later, I can show you the code, and all that we have been doing for the various kinds of drivers.” And they hurried towards their classroom.

Friday, 24 June 2016

Collection Of 51 Free eBooks On Python Programming

To use with Raspberry Pi or to program simulation software, here’s bringing 51 super helpful and free ebooks on Python programming that can help you become a python pro. Who knows you end up becoming the next big gaming programmer!
Atithya Amaresh

Author: Cody Jackson, 2013
An introduction to computer programming, using the easy, yet powerful, Python programming language. Python, a cross-platform language, lets you work quickly and efficiently, allowing you to concentrate on your work rather than the language.
Author: Brian Heinold
Publisher: Mount St. Mary’s University, 2013
Partly a tutorial and partly a reference book. I summarize information in tables and give a lot of short example programs. I also jump right into things and fill in background information as I go, rather than covering the background material first.
Author: Steven Thurlow
Publisher: Wikibooks, 2013
Contents of Beginner’s Python Tutorial: Installing Python; Very Simple Programs; Variables, Scripts; Loops, Conditionals; Functions; Tuples, Lists, Dictionaries; for Loop; Classes; Importing Modules; File I/O; Exception Handling.
Author: David Beazley, Brian K. Jones
Publisher: O’Reilly Media, 2013
This cookbook is for experienced Python programmers who want to focus on modern tools and idioms. You’ll find complete recipes for more than a dozen topics, covering the core Python language as well as tasks common to a wide variety of applications.
Author: Al Sweigart, 2013
The book teaches complete beginners how to program in the Python programming language. The book features the source code to several ciphers and hacking programs for these ciphers. The programs include the Caesar cipher, transposition cipher, etc.
Author: Nathan Yergler
Publisher: PyCon, 2013
Effective Django development means building applications that are testable, maintainable, and scalable. After reading this book you should have an understanding of how Django’s pieces fit together and how to use them to engineer web applications.
Author: Harry Percival
Publisher: O’Reilly Media, 2013
Test-Driven Development with Python focuses on web development, with some coverage of JavaScript. This book uses a concrete example — the development of a website, from scratch — to teach the TDD methodology and how it applies to web programming.
Author: Jan Bodnar
Publisher: ZetCode, 2013
PyQt4 is a toolkit for creating GUI applications. It is a blending of Python language and the successful Qt library. This tutorial is suited for beginners and intermediate programmers. You will learn to program non trivial PyQt4 applications.
Author: Hannes Röst, et al.
Publisher: Wikibooks, 2013
This book describes Python, an open-source general-purpose interpreted programming language, available for all Platforms. Python is a high-level, structured programming language that can be used for a wide variety of programming tasks.
Author: EuroScipy tutorial team, 2012
Teaching material on the scientific Python ecosystem, a quick introduction to central tools and techniques. The different chapters each correspond to a 1 to 2 hours course with increasing level of expertise, from beginner to expert.
Author: John C. Lusth
Publisher: The University of Alabama, 2012
Contents: Starting Out; Literals; Combining Literals; Precedence and Associativity; Variables; Assignment; Conditionals; Functions; Python Programs and Using Files; Input and Output; More about Functions; Scope; Loops; Lists; Recursion; etc.
Author: Lennart Regebro
Publisher: Colliberty 2011
Porting to Python 3 doesn’t have to be daunting. This book guides you through the process of porting your Python 2 code to Python 3. Using plenty of code examples is takes you cross the hurdles and shows you the new Python features.
Author: Kevin Sheppard, 2012
Python is a widely used general purpose programming language, which happens to be well suited to Econometrics and other more general purpose data analysis tasks. These notes provide an introduction to Python for a beginning programmer.
Author: Hans Petter Langtangen
Publisher: Springer, 2009
With a primary focus on examples and applications of relevance to computational scientists, this useful book shows computational scientists how to develop tailored, flexible, and human-efficient working environments built from small scripts.
Author: W.J. Turkel, A. Crymble, A. MacEachern
Publisher: NiCHE, 2010
This book is a tutorial-style introduction to programming in Python for practicing historians (and other humanists). We assume that you’re starting out with no prior programming experience and only a basic understanding of computers.
Author: Jan Erik Solem
Publisher: O’Reilly Media, 2012
The idea behind this book is to give an easily accessible entry point to hands-on computer vision with enough understanding of the underlying theory and algorithms to be a foundation for students, researchers and enthusiasts.
Author: Rahul Verma, Chetan Giridhar
Publisher: Testing Perspective, 2011
This book is about learning design patterns with Python language. If you are new to design patterns, this text provides the first building blocks. If you are interested in design of test automation frameworks, this book will be very useful.
Author: Al Sweigart, 2012
This is a programming book that covers the Pygame game library for the Python programming language. Each chapter gives you the complete source code for a new game and teaches the important programming concepts from these examples.
Author: Richard L. Halterman
Publisher: Southern Adventist University, 2011
The focus is on introducing programming techniques and developing good habits. Our approach avoids some more esoteric features of Python and concentrates on the programming basics that transfer directly to other imperative programming languages.
Author: Jeffrey Elkner, Allen B. Downey, Chris Meyers
Publisher: Green Tea Press, 2012
This book is an introduction to computer science using the Python programming language. It covers the basics of programming, including variables, functions, control flow, program debugging. Later chapters cover basic algorithms and data structures.
Author: Steven Bird, Ewan Klein, Edward Loper
Publisher: O’Reilly Media, 2009
This book offers a highly accessible introduction to natural language processing, the field that supports a variety of language technologies. With it, you’ll learn how to write Python programs that work with large collections of unstructured text.
Author: Zed A. Shaw, 2011
This is a very beginner book for people who want to learn to code. If you can already code then the book will probably drive you insane. It’s intended for people who have no coding chops to build up their skills before starting a more detailed book.
Author: Allen B. Downey
Publisher: Green Tea Press, 2011
Think Stats is an introduction to Probability and Statistics for Python programmers. This new book emphasizes simple techniques you can use to explore real data sets and answer interesting statistical questions. Basic skills in Python are assumed.
Author: Charles Severance
Publisher: PythonLearn, 2010
The goal of this book is to provide an Informatics-oriented introduction to programming. The primary difference between a computer science approach and the Informatics approach taken in this book is a greater focus on using Python.
Author: Jody Scott Ginther
Publisher: toonzcat.com, 2010
This book is meant to help you begin learning the basics of Python programming version 3 or later. It is a brief introduction to Python. The author attempts to be as brief as possible to get the new programmer into programming as fast as possible.
Author: Jeffrey Elkner, at al., 2010
The goal of this book is twofold: to teach you how to program in Python; and to teach you to think like a computer scientist. This way of thinking combines some of the best features of mathematics, engineering, and natural science.
Author: Mark Guzdial
Publisher: Georgia Institute of Technology, 2002
Guzdial introduces programming as a way of creating and manipulating media — a context familiar to today’s readers. Starts readers with actual programming early on. Puts programming in a relevant context (Computing for Communications).
Author: Jason R Briggs
Publisher: Lulu.com, 2007
Snake Wrangling for Kids is a printable electronic book, for children 8 years and older, who would like to learn computer programming. It covers the very basics of programming, and uses the Python 3 programming language to teach the concepts.
Author: A. Holovaty, J. Kaplan-Moss
Publisher: Apress, 2007
Django is the Python–based equivalent to the Ruby on Rails web development framework. The authors show you how they use this framework to create award–winning web sites. This is the first edition of the Django Book which covers 0.96 version.
Author: Katja Schuerer, Catherine Letondal
Publisher: Pasteur Institute, 2008
This course is designed for biologists who already have some programming knowledge in other languages. The focus is on biological examples that are used throughout the course, as well as the suggested exercises drawn from the field of biology.
Author: Dave Kuhlman, 2003
This document contains discussions of several advanced topics that are of interest to Python programmers: regular expressions, unit tests, extending and embedding Python, parsing, GUI applications, guidance on packages and modules.
Author: Dave Kuhlman, 2008
This document is a syllabus for a first course in Python programming. It contains an introduction to the Python language, instruction in the important features of the language, and practical exercises in the use of those features.
Author: Chris Meyers, 2004
This collection is a presentation of several small Python programs. They are aimed at intermediate programmers – people who have studied Python and are fairly comfortable with basic recursion and object oriented techniques.
Author: Fredrik Lundh, Matthew Ellis
Publisher: PythonWare, 2002
The Python Imaging Library adds image processing capabilities to your Python interpreter. This library provides extensive file format support, an efficient internal representation, and fairly powerful image processing capabilities.
Publisher: Python Software Foundation, 2008
This tutorial introduces the reader informally to the basic concepts of the Python language and system. It introduces many of Python’s most noteworthy features, and will give you a good idea of the language’s flavor and style.
Author: Richard Gruet, 2007
This reference collections cover references for python version 1.52 to version 2.7. It covers invocation options, environment variables, lexical entities, basic types and their operations, advanced types, statements, iterators, generators, descriptors, decorators, built-in functions, built-in exceptions, and more.
Author: Steven F. Lott, 2008
The book will help you build basic programming skills. It is organized in a way that builds up the language in layers from simple concepts to more advanced features. Programming exercises are provided to encourage further exploration of each layer.
Author: Guido Van Rossum
Publisher: Network Theory Ltd., 2003
The definitive language reference for Python. It describes the syntax of Python and its built-in datatypes. It covers advanced topics, and is suitable for readers who are familiar with the details and rules of the Python and its object system.
Author: Bruno R. Preiss, 2004
The primary goal of this book is to promote object-oriented design using Python and to illustrate the use of the emerging object-oriented design patterns. The book shows how these patterns are used to create good software designs.
Author: Allen Downey
Publisher: Green Tea Press, 2008
A concise introduction to software design using Python. Intended for people with no programming experience, this book starts with the most basic concepts and gradually adds new material. The goal is to teach you to think like a computer scientist.
Author: Alan Gauld
Publisher: Addison-Wesley Professional, 2000
Book for hobbyists, self-starters, and all who want to learn the art of computer programming in Python. Data types and variables, debugging, and namespaces are covered. Also includes sample applications that illustrate ideas and techniques in action.
Author: K. Schuerer, et al.
Publisher: Pasteur Institute, 2008
This text teaches programming concepts to biologists. It is aimed at people who are not professional computer scientists, but who need a better control of computers for their own research. This course is part of a course in informatics for biology.
Author: David Mertz
Publisher: Addison-Wesley Professional, 2003
A hands-on tutorial that teaches how to accomplish numerous text processing tasks using Python. Filled with examples, the book provides efficient solutions to specific text processing problems and strategies for all text processing challenges.
Author: Swaroop C H, 2008
This is a tutorial and a guide on Python programming for absolute beginners. If all you know about computers is how to save text files, then this is the book for you. The book is updated for the upcoming Python 3.0 language.
Author: Josh Cogliati
Publisher: Wikibooks, 2005
This free tutorial is designed to be an introduction to the Python programming language. The guide is for someone with no programming experience. The author attempts to teach programming by reading code and writing code.
Author: Steven F. Lott, 2008
This book is a complete presentation of the Python for professional programmers who need to learn the language. The author leads you from a tiny, easy to understand subset of statements to the entire Python language and all built-in data structures.
Author: Fredrik Lundh
Publisher: O’Reilly, 2001
A large collection of useful Python scripts, the best parts of comp.lang.python newsgroup messages, plus hundreds of new scripts. The text covers the standard library, the examples should work on most platforms and Python versions.
Author: Boudewijn Rempt
Publisher: OpenDocs, LLC, 2002
This book covers application development using the library extension PyQt, which forms the basis for GUI programming. First part explains concepts using small examples, in the second part the author develops a complete, complex application.
Author: Mark Pilgrim
Publisher: Apress, 2004
This is a book for experienced programmers, a hands-on guide to the Python language. Each chapter starts with a complete code sample, picks it apart and explains the pieces, and then puts it all back together in a summary at the end.
Author: Guido van Rossum
Publisher: Network Theory Ltd., 2006
This book is an introduction to Python, an easy to learn, powerful programming language. The tutorial explains the basics of the Python, it does not cover every single feature of the language, but introduces the most noteworthy features.
Author: Albert Sweigart
Publisher: Albert\Sweigart, 2008
A programming book for kids interested in learning to program their own computer games with python, a serious computer language which is used by professional programmers also. The book explains programming principles from the source code examples.

Thursday, 23 June 2016

Redirection and Pipes

3.1 Redirection  

Most processes initiated by Linux commands write to the standard output (that is, they write to the terminal screen), and many take their input from the standard input (that is, they read it from the keyboard). There is also the standard error, where processes write their error messages, by default, to the terminal screen.
We have already seen one use of the cat command to write the contents of a file to the screen.
Now type cat without specifying a file to read
% cat
Then type a few words on the keyboard and press the [Return] key.
Finally hold the [Ctrl] key down and press [d] (written as ^D for short) to end the input.
What has happened?
If you run the cat command without specifying a file to read, it reads the standard input (the keyboard), and on receiving the "end of file" (^D), copies it to the standard output (the screen).
In Linux, we can redirect both the input and the output of commands.

3.2 Redirecting the Output  

We use the > symbol to redirect the output of a command. For example, to create a file called list1 containing a list of fruit, type  
% cat > list1
Then type in the names of some fruit. Press [Return] after each one.
pear
banana
apple
^D (Control D to stop)
What happens is the cat command reads the standard input (the keyboard) and the > redirects the output, which normally goes to the screen, into a file called list1
To read the contents of the file, type
% cat list1

Exercise 3a

Using the above method, create another file called list2 containing the following fruit: orange, plum, mango, grapefruit. Read the contents of list2
The form >> appends standard output to a file. So to add more items to the file list1, type
% cat >> list1
Then type in the names of more fruit
peach
grape
orange
^D (Control D to stop)
To read the contents of the file, type
% cat list1
You should now have two files. One contains six fruit, the other contains four fruit. We will now use the cat command to join (concatenate) list1 and list2 into a new file called biglist. Type
% cat list1 list2 > biglist
What this is doing is reading the contents of list1 and list2 in turn, then outputting the text to the file biglist
To read the contents of the new file, type
% cat biglist

3.3 Redirecting the Input  

We use the < symbol to redirect the input of a command.
The command sort alphabetically or numerically sorts a list. Type
% sort
Then type in the names of some vegetables. Press [Return] after each one.
carrot
beetroot
artichoke
^D (control d to stop)
The output will be
artichoke
beetroot
carrot
Using < you can redirect the input to come from a file rather than the keyboard. For example, to sort the list of fruit, type
% sort < biglist
and the sorted list will be output to the screen.
To output the sorted list to a file, type,
% sort biglist > slist
Use cat to read the contents of the file slist

3.4 Pipes

To see who is on the system with you, type
% who
One method to get a sorted list of names is to type,
% who > names.txt
% sort names.txt
This is a bit slow and you have to remember to remove the temporary file called names when you have finished. What you really want to do is connect the output of the who command directly to the input of the sort command. This is exactly whatpipes do. The symbol for a pipe is the vertical bar |
For example, typing
% who | sort
will give the same result as above, but quicker and cleaner.
To find out how many users are logged on, type
% who | wc -l

Exercise 3b

lpr -Plabprinter file is the command to print a postscript file to the printer labprinter.
Using pipes, print all lines of list1 and list2 containing the letter 'p', sort the result, and print to the printer labprinter.

Summary

command > fileredirect standard output to a file
command >> fileappend standard output to a file
command < fileredirect standard input from a file
command1 | command2pipe the output of command1 to the input of command2
cat file1 file2 > file0concatenate file1 and file2 to file0
sortsort data
wholist users currently logged in
lpr -Pprinter fileprint file to named printer