python

How top 10 programming languages work

TIOBE index: July 2014

I am programming for more than 10 years. I realized that, when I found my first, personal home page (written in PHP) on my hard drive. Some files have last modification date: May 2003. Since that time I was working (or playing) with 9 of top 10 programming language from TIOBE Index list. As I mentioned, I started with PHP, then I learnt a little bit JavaScript (to create fancy menus or ‘secret pages’ on my website). In high school I started learning C++. At the University I learnt C, more C++ and Java. Then I learnt C# on my own, and it is still my favorite language. Along with C# I learnt ASP.NET and T-SQL (to be able to create websites with databases). I also learnt Python and Objective-C. Former became useful when I started my Research Assistant Job.

More experience I gain, more interested I am in how things work underneath the code level. In this post I want to provide short description of the Top 10 programming languages (by TIOBE Index), how they work and their super short history.

PHP

Interpreted language created by Rasmus Lerdorf. Originally, collection of Perl scripts, rewritten to C for performance reasons, ability to work with web-forms and communicate with databases. Most popular interpreter: Zend Engine. Since PHP 4, the PHP parser compiles input to produce bytecode for processing by the Zend Engine. Facebook developed two PHP interpreters: HipHop (transforms the PHP scripts into C++ code and then compiles) and HipHop Virtual Machine (converts PHP into a high-level bytecode, which is then translated into x86-64 machine code dynamically). Recommended book: PHP and MySQL Web Development.

JavaScript

Dynamic, interpreted and prototype-based language. JavaScript is superset of ECMAScript standard. Usually used as part of web browser, but also in server-side (Node.js) or even for desktop and mobile application development. Current browsers perform just-in-time compilation. There are many JavaScript engines (interpreters): Rhino (written in Java), SpiderMonkey (written in C, first JS engine), V8 (written C++, by Google, used in Google Chrome), Chakra (performs JIT compilation on a separate CPU core, parallel to the web browser; created by Microsoft, used in Internet Explorer) etc. Recommended book: JavaScript: The Good Parts.

C/C++

C is compiled, static type language created by Denis Ritchie. It influenced many other languages, e.g.: C++, C#, Java, JavaScript, Objective-C, Perl, PHP, Python, and even Unix’s C Shell. C++ is nearly a superset of C, with object oriented features inspired by Simula language, created by Bjarne Stroustrup. C and C++ are compiled to machine specific code, thus platform specific compiler is needed. Most popular compilers: GCC and Visual C++. C++ evolve, Modern C++ (currently C++ 11 and vNext: C++ 14) introduces: smart pointers, for each (range for), lambda functions and much more. Recommended book: Ivor Horton’s Beginning Visual C++ 2012The C Programming Language (by Brian Kernighan and Denis Ritchie)  used to be one of the most popular programming books ever.

Java

C-based language created by James Gosling, which introduces automatic memory management by Garbage Collector. Java is compiled to bytecode (.class files), which runs on JVM (Java Virtual Machine). Its main design goal is “Write Once, Run Anywhere” (WORA). The same code can be compiled to the same bytecode on different machines. Then JVM (platform-specific) translates bytecode to particular machine code during run-time (Just-in-Time (JIT) compilation). Java applets can run in web browser. My personal recommendation for Java book is Hortsmann’s Core Java Volume I – Fundamentals and Volume II – Advanced Features. Well known Java book is Bruce Eckel’s Thinking in Java.

C#

C# is Microsoft’s version of Java language created by Anders Hejlsberg. However, C# is closer to C++ in case of design and syntax. It is compiled to Intermediate Language (equivalent of bytecode in Java) and runs on Common Language Runtime (equivalent of JVM). C# also uses Garbage Collector. It has many features not existing in Java like rich native interoperability, query language (LINQ) or dynamic type. For more, check Comparison of C# and Java. Book for quick start: Sams Teach Yourself Visual C# 2010 in 24 Hours recommended by Eric Lippert (former developer of C# compiler). To dive in, check: C# 5.0 in a Nutshell: The Definitive Reference by Joseph Albahari and Ben Albahari, C# in Depth by Jon Skeet and CLR via C# by Jeffrey Richter.

Python

Dynamic, interpreted language, created by Guido van Rossum. Most popular implementation: CPython (implemented in C). It compiles Python programs to intermediate code (.pyc files) and runs them on Virtual Machine. There are many other implementations, e.g. Jython, which compiles Python to Java Bytecode (.class files). Python is much more expressive language than C or Java. Some constructs, which needs 10 lines of C code or 3 lines of Java code, requires only 1 line in Python (check reverse words example in my post about Python). Python uses whitespace indentation, rather than curly braces or keywords, to delimit blocks. To get started I recommend Google’s Python Class (videos section) created by Nick Parlante. Other resources: Python tutorial and Dive into Python. You can also check my post Python jump start for more details.

Objective-C

Strict superset of C language with object-oriented capabilities inspired by Smalltalk. Every C program can be compiled by Objective-C compiler, C code can be included within Objective-C code. Most popular compilers: Clang and LLVM. Instead of calling methods (like in C++: obj->method(param)), messages to objects are being sent ([obj method:param]) and resolved at runtime (not at compile time like in C++). There is no Garbage Collector (which works at run-time) in Objective-C, but instead Automatic Reference Counting (compile-time feature) is used. Objective-C is the main programming language used by Apple for the OS X and iOS. However, in this year Apple announced plans to replace Objective-C with new Swift language. Resources to learn Objective-C can be found in my other post.

Transact-SQL

Declarative language for managing data held in a relational database management system (RDBMS). Created by Microsoft (for MS SQL Sever), T-SQL is an extension to SQL, which makes it Turing complete. It adds to SQL: procedural programming, local variables, functions for string and date processing, mathematics etc. and allows FROM clause in DELETE and UPDATE statements (which allows joins to be included). T-SQL (and SQL) query differs from program in such a way that they just tell what to do, not how to do that. Figuring out how to execute query is a role of query analyzer. Check Understanding how SQL Server executes a query.  To learn T-SQL, you should learn SQL first. Recommended tutorial: T-SQL Step by Step Course (video tutorial). Recommended books: Itzik Ben-Gan’s Microsoft SQL Server 2012 T-SQL Fundamentals and other his books.

(Visual) Basic (.NET)

Compiled language created by Microsoft. Compiles to native language or P-Code and uses the Microsoft C++ compiler to generate the executable. It derives from BASIC. Similar to Python: tabs and new lines are used to delimit blocks. VB (under Visual Studio) allows to create GUI using drag-and-drop technique. Last version (VB6) was released in 1998 and is abandoned in favor of VB.NET, which introduces many features (present also in C#), but still supported, even on Windows 8. Check Comparison of VB and VB.NET. Both (VB.NET and C#) run on the same run time (CLR). More details can be found here. Recommended tutorials to get started: Visual Basic Fundamentals: Development for Absolute Beginners by Bob Tabor and Visual Basic .NET Tutorials. Good reference is Visual Basic at Wikibooks. To get VB syntax highlighting in SublimeText, this package works.

Summary

I read about all 10 languages in Wikipedia before writing this post. I was surprised how much I could learn in 5-10 minutes reading. I don’t know why, but I feel much better now, when I refreshed and organized my knowledge. If you want to start learning about new programming language, I recommend you to read about it on Wikipedia first. Even more, read about all languages you are using now on Wikipedia as well. It gives you great, high-level overview. Now, Wikipedia will be my start point of research about “some new thing”.


Interprocess communication between Python and Java

For my Research work I need to communicate between Python and Java/Scala application.

The first idea was to use WebSockets. It is universal and flexible. Additionally communication is easy. There are many ways to implement it:

Implementation would be easy, but we are using customized version of Python, which is included in GNAT Programming Studio. Some libraries (required for WebSockets) are excluded (because of legal reasons) and we wouldn’t be able to redistribute it along with missing packages. Thus, we decide to implement interprocess communication through pipes.

From Python to Java

The communication is one direction: from Python app to Java app. Every message is one line of text (ultimately: json format). E.g.: “text\r\n”. To close communication, we send message: “x\r\n”.

This is sample Java app, which write received messages to file result.txt:

import java.io.*;

public class MyClass {
	public static void main(String[] args) {
		try {
			BufferedReader bufferRead = new BufferedReader(new InputStreamReader(System.in));
			PrintWriter writer = new PrintWriter("result.txt", "UTF-8");
			String s = bufferRead.readLine();
			while(s.equals("x")==false) {
				writer.println(s);
				s = bufferRead.readLine();
			}
			writer.close();
		} catch(IOException e) {
			e.printStackTrace();
		}
	}
}

And here is sample Python app, which sends messages to Java app:

#!/usr/bin/python

import subprocess

p = subprocess.Popen(["java", "MyClass"], stdin=subprocess.PIPE)
p.stdin.write("First line\r\n")
p.stdin.write("Second line\r\n")
p.stdin.write("x\r\n") # this line will not be printed into the file

From Java to Python

We can also communicate in opposite direction: from Java app to Python app. To do that we just need to change reading to writing in Java app and writing to reading in Python app.

Change in Java app is simple: we just write to standard output.

import java.io.*;

public class MyClass2 {
	public static void main(String[] args) throws InterruptedException{
		System.out.println("First msg");
		System.out.println("Second msg");
		System.out.println("x");
	}
}

In Python app we need to create stdout pipe and read instead of write:

#!/usr/bin/python

import subprocess

p = subprocess.Popen(["java", "MyClass2"], stdout=subprocess.PIPE)
line = p.stdout.readline()
while(line != "x\n"):
	print line
	line = p.stdout.readline()

In this scenario, both processes are managed by Python app. You can also do it in opposite direction and invoke Python app from Java app.


PyGTK, Multithreading and progress bar

Multithreading in Python seems to be very simple. Let’s look at the example:

import threading
import time

global_var = 0

class ThreadClass(threading.Thread):
	def run(self):
		global global_var
		while True:
			print global_var

class ThreadClass2(threading.Thread):
	def run(self):
		global global_var
		while True:
			time.sleep(1)
			global_var += 1

t1 = ThreadClass()
t1.start()

t2 = ThreadClass2()
t2.start()

t1.join()
t2.join()

Thread 1 is printing the global_var and Thread 2 is incrementing it by 1, every second.

Recently I needed to use threads in PyGTK to display progress bar. Off course the state of progress bar was dependent on the other thread. It had to be updated after each subtask call. This is the code:

import threading
import time
import gtk

def init_progressbar():
    main_box = gtk.VBox()
    progressbar = gtk.ProgressBar()    
    progressbar_box = gtk.HBox(False, 20)
    main_box.pack_start(progressbar_box, False, False, 20)
    progressbar_box.pack_start(progressbar)    
    info_box = gtk.VBox()
    main_box.pack_start(info_box, False, False, 10)    
    info_label = gtk.Label("Running...")
    info_box.pack_start(info_label)
    return main_box, progressbar, info_label


def run_tasks(pb, info_label):
    task_list = ['task1', 'task2', 'task3', 'task4']
    task_no = 0
    for task in task_list:
        pb.set_fraction(float(task_no)/len(task_list))
        run_task(2, task, info_label)        
        task_no += 1
    pb.set_fraction(float(task_no)/len(task_list))
    info_label.set_text('Finished')


def run_task(delay, task, info_label):
    info_label.set_text(task + ' is running...')
    time.sleep(delay)   # simulation of running the task


win_pb, pb, info_label = init_progressbar()
win = gtk.Window()
win.set_default_size(400,100)
win.add(win_pb)
win.show_all()
win.connect("destroy", lambda _: gtk.main_quit())
t = threading.Thread(target=run_tasks, args=(pb,info_label))
t.start()
gtk.main()

And…it was not working. The progress bar (and the label) did not get updated after task calls. Why?

It was caused by the fact, that PyGTK is not thread safe. Fortunately we have function threads_init() in gobject module, which can handle it. We just need to call gobject.threads_init() before first thread creation. Corrected (working) code looks like that:

import threading
import time
import gobject
import gtk

def init_progressbar():
    main_box = gtk.VBox()
    progressbar = gtk.ProgressBar()    
    progressbar_box = gtk.HBox(False, 20)
    main_box.pack_start(progressbar_box, False, False, 20)
    progressbar_box.pack_start(progressbar)    
    info_box = gtk.VBox()
    main_box.pack_start(info_box, False, False, 10)    
    info_label = gtk.Label("Running...")
    info_box.pack_start(info_label)
    return main_box, progressbar, info_label


def run_tasks(pb, info_label):
    task_list = ['task1', 'task2', 'task3', 'task4']
    task_no = 0
    gobject.threads_init()
    for task in task_list:
        pb.set_fraction(float(task_no)/len(task_list))
        run_task(2, task, info_label)        
        task_no += 1
    pb.set_fraction(float(task_no)/len(task_list))
    info_label.set_text('Finished')


def run_task(delay, task, info_label):
    info_label.set_text(task + ' is running...')
    time.sleep(delay)   # simulation of running the task


win_pb, pb, info_label = init_progressbar()
win = gtk.Window()
win.set_default_size(400,100)
win.add(win_pb)
win.show_all()
win.connect("destroy", lambda _: gtk.main_quit())
t = threading.Thread(target=run_tasks, args=(pb,info_label))
t.start()
gtk.main()

I wanted also to be able to cancel program computation, by the cancel button. To do that I needed shared variable to be a status flag informing, whether computation should be cancelled or continued. It had to be accessible in task loop and cancel event.

In Python there are mutable and immutable objects. Mutable are passed by reference, but immutable – by value. More about that on this StackOverflow question. Boolean type is immutable, but there is a way around this: use list (mutable type) with one element (boolean variable). I know it is awful solution, but I did not find any better. If you know one, let me know!

This is the app with cancel button:

import threading
import time
import gobject
import gtk

def init_progressbar():
    main_box = gtk.VBox()
    progressbar = gtk.ProgressBar()    
    progressbar_box = gtk.HBox(False, 20)
    main_box.pack_start(progressbar_box, False, False, 20)
    progressbar_box.pack_start(progressbar)    
    info_box = gtk.VBox()
    main_box.pack_start(info_box, False, False, 10)    
    info_label = gtk.Label("Running...")
    info_box.pack_start(info_label)
    cancel_box = gtk.HBox()
    info_box.pack_start(cancel_box)
    cancel_button = gtk.Button("Cancel")
    cancel = [False]
    cancel_button.connect("clicked", cancel_counting, info_label, cancel)
    cancel_box.pack_start(cancel_button, False, False, 20)
    return main_box, progressbar, info_label, cancel


def run_tasks(pb, info_label, cancel):
    task_list = ['task1', 'task2', 'task3', 'task4']
    task_no = 0
    gobject.threads_init()
    for task in task_list:
        if cancel[0]:       
            pb.set_fraction(0)
            info_label.set_text('Canceled')
            return
        pb.set_fraction(float(task_no)/len(task_list))
        run_task(2, task, info_label)        
        task_no += 1
    pb.set_fraction(float(task_no)/len(task_list))
    info_label.set_text('Finished')


def run_task(delay, task, info_label):
    info_label.set_text(task + ' is running...')
    time.sleep(delay)   # simulation of running the task


def cancel_counting(widget, info_label, cancel):
    cancel[0] = True
    info_label.set_text("Cancelling...")


win_pb, pb, info_label, cancel = init_progressbar()
win = gtk.Window()
win.set_default_size(400,100)
win.add(win_pb)
win.show_all()
win.connect("destroy", lambda _: gtk.main_quit())
t = threading.Thread(target=run_tasks, args=(pb,info_label, cancel))
t.start()
gtk.main()

To make above code working on your machine you need to have Python and PyGTK installed. To find out the details, how to install Python and/or PyGTK check my Python jump start post.


Research Assistant Job – Project Sireum

Since January 2013 I am Master student at Kansas State University. I am also Research Assistant at SAnToS lab.

Bakar toolset

Since the beginning I am working on project Sireum. It is focused on toolset for analyzing Spark/Ada programs: Bakar. Spark is subset of Ada programming language extended by code contracts. More info here. Bakar contains two main tools: Alir (information flow analysis tool) and Kiasan (verification tool for code contracts).

The Kiasan tool is based on symbolic execution. Simply saying: it checks all possible code paths and discover possible issues (like variable overflow, side effects etc.). More info about Spark contracts can be found in Tokeneer Discovery Tutorial. Comprehensive description of Kiasan in contained in this paper.

Kiasan plugin for GPS

My first task was to integrate Kiasan tool (Java app) with GPS (GNAT Programming Studio).

I have never working on IDE plugins before. It was something totally new for me. That’s why I was excited.

Plugins for GPS are written in PyGTK (Python graphic library). First I needed to learn Python and PyGTK. Then I followed GPS documentation (especially chapters 16 and 18).

The sample flow of running Kiasan plugin is as follows:

  • right click on method (which you want to analyze) and choose ‘Run Kiasan’
  • handle this event and get method name and package name (will be needed to run Kiasan)
  • run external Java program (Kiasan.jar) from Python code (it has command line interface, so I just needed to execute subprocess.call method), which generates JSON report
  • parse JSON report and display it in the PyGTK window attached to the GPS
  • add code highlighting, dependent on chosen method or case (in report window by double-click)

GPS along with Kiasan plugin is available in Sireum distribution, which can be downloaded from http://www.sireum.org/download.

In the demo below I am showing how to install Sireum distribution and GPS with Kiasan plugin on Windows.

By the way: Sireum Project is Open Source on github!

Do I like GRA job? Yes. It is very nice experience. I am able to work on Mac. I didn’t have this opportunity before (I wouldn’t buy it soon probably). I was able to learn Python (and PyGTK). I learn Design by Contract. I learn how to create plugin for IDE (GNAT Programming Studio). We have continues integration server – Jenkins: I learn how to manage it. I explore the World of ‘High Assurance Software’ and Ada programming language, which is most common in such environment. It is totally different than working as software developer in some company.

Recently I started working on second SAnToS’ project: Medical Device Coordination Framework. Another interesting thing. But I will tell you about it in another post.

Interesting fact: Ada programming language was named after Ada Lovelace (1815–1852), who is credited as being the first computer programmer.


Python jump start

In my current job (Research Assistant at SAnToS lab) I had to learn Python. I was very happy because of that. It gaves me an opportunity to get familiar with one of the most popular programming languages nowadays.

I was very lucky to find awesome Google’s Python Class by Nick Parlante. It is great! If you want to start programming with Python or just learn it for fun, start with this tutorial!

As a supplement to above course you can read some more detailed tutorial. I went through two: Learn Python The Hard Way and Tutorial from Python documentation. However if you already know some other programming language(s), your should learn during development. Python contains almost all common features of programming languages such as if/else, loops, exceptions, functions, classes etc. I said ‘almost’, because there is e.g. no switch instruction. However to check things like that there is very well written documentation. It contains a lot of examples. The main difference between other popular languages like (C, C# or Java) and Python is that there is no semicolons. We use colons and indentation instead.

if number > 0:
  print "This is natural number."
else:
  print "This is not natural number."

Python is dynamic, strongly typed programming language. It means type checking occurs during the run time, instead of compilation time. Programming in Python is a real pleasure. Sometimes you can explicitly put your mind into the code. That is because of high level of abstraction. E.g. file operations are so simple and intuitive. You do not need to remember any StreamReaders or BufferedReaders and bunch of functions for simple I/O operations. Below example reads content of file.

f = open('file.txt')
f.read()
f.close()

Cool feature is the possibility to call functions explicitly on string. Like that:

"jakub".upper()

There is a lot of implemented (widely used) functions in Python. As a comparison, let’s see how to reverse words in a sentence using C, Java and Python.

In C:

void reverse_words(char *sentence)
{
   char *start = sentence;
   char *end = sentence;

   while (*end != '\0') {
      ++end;
   }
   --end;

   reverse_chars(start, end);
   
   while (*start != '\0') {
      for (; *start != '\0' && *start == ' '; start++) ;
      for (end=start; *end != '\0' && *end != ' '; end++) ;
      --end;
      reverse_chars(start, end);
      start = ++end;
   }
}

void reverse_chars(char *left, char *right)
{
   char temp;

   while( left < right) {
      temp = *left;
      *left = *right;
      *right = temp;
      ++left;
      --right;
   }
}

In Java:

public string ReverseWords(string sentence)
{
  string[] words =  sentence.split(" ");
  string rev = "";
  for(int i = words.length - 1; i >= 0 ; i--)
  {
    rev += words[i] + " ";
  }
  return rev;
}

In Python:

def reverse_words(sentence):
  return " ".join(reversed(sentence.split(" ")))

That’s why Python is good for Rapid Development.

I am also using PyGTK (graphic library for Python) in my work. There is a great tutorial Python GTK on youtube! PyGTK requires very less code than e.g. C# to create some simple application. We do not to have tons of generated code when we start. We create application from scratch. Look at below Hello World example.

import pygtk
pygtk.require('2.0')
import gtk

class HelloWorld:
    def hello(self, widget, data=None):
        print "Hello World"

    def destroy(self, widget, data=None):
        gtk.main_quit()

    def __init__(self):
        self.window = gtk.Window(gtk.WINDOW_TOPLEVEL)
        self.window.connect("destroy", self.destroy)
        self.window.set_border_width(10)
        self.button = gtk.Button("Hello World")
        self.button.connect("clicked", self.hello, None)
        self.window.add(self.button)
        self.button.show()
        self.window.show()

    def main(self):
        gtk.main()

if __name__ == "__main__":
    hello = HelloWorld()
    hello.main()

The result is the window with button ‘Hello World’. When you click the button, then ‘Hello World’ will be printed on console. All of that with 22 lines of code (I do not count white lines).

Hello Python

If you don’t know python yet, I encourage you to try it. Programming in python requires a little bit different way of thinking. It also allows you to look at the programming from the different perspective.

Python installation is easy on all operating systems and you can find it in google. To install PyGTK in Windows you can use all in-one installer. There is also all-in-one installer for Mac. PyGTK is included in most Linux distributions, so you won’t need to install it if you are using Linux.