Understanding the History of Linux

Understanding theHistory of Linux Tero Roponen

Contents • My background • About the research • Current status • Examples • Future work

My background (1/2) Tero Roponen (1978 - ????) • Bachelor of Engineering in Programming Techniques Jyväskylä Polytechnic, 2003 • Master of Science in Software Engineering University of Jyväskylä, 2007 • PhD student since 2008

My background (2/2) • Linux user since 1997 • Some minor contributions to Open Source projects with varying success: • “Thanks for the report.” (GIT) • “Thanks, applied.” (Cogito) • “Whoa. Impressed.” (Linux kernel) • “tero is in the real pro category, lots to learn from him” (Tux3 filesystem)

About the research Free/Libre/Open Source Software (FLOSS) Evolution: Information for Program Comprehension In Linux and other Systems

Goals • Identify fundamental problems in today’s software (i.e. Linux)

Goals • Identify fundamental problems in today’s software (i.e. Linux) • not ordinary bugs but something more… fundamental • e.g. why a filesystem can’t handle files bigger than 2 gigabytes

Goals • Identify fundamental problems in today’s software (i.e. Linux) • not ordinary bugs but something more… fundamental • e.g. why a filesystem can’t handle files bigger than 2 gigabytes • Find design decisions behind these problems

Goals • Identify fundamental problems in today’s software (i.e. Linux) • not ordinary bugs but something more… fundamental • e.g. why a filesystem can’t handle files bigger than 2 gigabytes • Find design decisions behind these problems • e.g. “2 gigabytes should be enough for all”

Goals • Identify fundamental problems in today’s software (i.e. Linux) • not ordinary bugs but something more… fundamental • e.g. why a filesystem can’t handle files bigger than 2 gigabytes • Find design decisions behind these problems • e.g. “2 gigabytes should be enough for all” • Try to learn from the past mistakes

Goals • Identify fundamental problems in today’s software (i.e. Linux) • not ordinary bugs but something more… fundamental • e.g. why a filesystem can’t handle files bigger than 2 gigabytes • Find design decisions behind these problems • e.g. “2 gigabytes should be enough for all” • Try to learn from the past mistakes • “64 bits is enough, or is it? Let’s make it 128”

Method (1/3) To locate a problem reading developer mailing-lists is required.

Method (1/3) To locate a problem reading developer mailing-lists is required. To understand the problem reading source code is required.

Method (1/3) To locate a problem reading developer mailing-lists is required. To understand the problem reading source code is required. To understand the source code understanding the design decisions is required.

Method (1/3) To locate a problem reading developer mailing-lists is required. To understand the problem reading source code is required. To understand the source code understanding the design decisions is required. To understand the design decisions reading developer mailing-lists is required.

Method (2/3) In practice: Study the problem at hand until you know what it does and why it happened. Look out for similarities between different problems. Repeat these a few times, then write a paper discussing the findings and the experience.

Method (3/3) Theoretical background • What is Free/Open Source Software? • How is it developed? • What is a patch?

Method (3/3) Theoretical background • What is Free/Open Source Software? • How is it developed? • What is a patch? There is no real answer to the last question in the literature – yet.

Current status (1/2) First article will be about patches: Patches as a Subject of Development & Research in the Context of Free/Libre/Open Source Software

Current status (2/2) First article • Work-in-progress • About 25 pages long From abstract: “In this article we take a deep look at patches: what they are, how they are being used and what information they can provide. With these issues in mind we propose a new framework for dealing with patches both in the development and research context.”

From: Tero Roponen <teanropo@cc.jyu.fi> Date: Sun, 31 Oct 2004 23:41:27 +0000 (-0800) Subject: [PATCH] Don't initialize /dev/pg0 to be always busy Without this patch my HP CD-Writer doesn't work. With this applied everything is fine. Signed-off-by: Tero Roponen <teanropo@cc.jyu.fi> Signed-off-by: Linus Torvalds <torvalds@osdl.org> --- diff --git a/drivers/block/paride/pg.c b/drivers/block/paride/pg.c index d73e3ec..8f28ff7 100644 --- a/drivers/block/paride/pg.c +++ b/drivers/block/paride/pg.c @@ -262,7 +262,7 @@ static void pg_init_units(void) int *parm = *drives[unit]; struct pg *dev = &devices[unit]; dev->pi = &dev->pia; - set_bit(0, &dev->access); + clear_bit(0, &dev->access); dev->busy = 0; dev->present = 0; dev->bufptr = NULL;

Example problem: the BKL (1/4) In the late 1990’s support for multiple processors was added to Linux. For the needed locking the Big Kernel Lock was introduced. Very simple idea: one CPU gets the lock and others wait. Doesn’t scale. Better alternatives exists. Must be removed. "The BKL can be subtle and evil, but as I brought it into the world I guess I must banish it ;)" - Alan Cox, 18 May 2008

Example problem: the BKL (2/4) The process of getting rid of BKL has been going on for years, and is still unfinished. “12 years after Linux has been converted to an SMP OS we still have 1300+ legacy BKL using sites. There are 400+ lock_kernel() critical sections and 800+ ioctls. They are spread out across rather difficult areas of often legacy code that few people understand and few people dare to touch.” - Ingo Molnar, 14 May 2008

Example problem: the BKL (3/4) “The biggest technical complication is that the BKL is unlike any other lock: it "self-releases" when schedule() is called. This makes the BKL spinlock very "sticky", "invisible" and viral: it's very easy to add it to a piece of code (even unknowingly) and you never really know whether it's held or not.” - Ingo Molnar, 14 May 2008 The problem: BKL is unlike any other lock. The solution: lots of hard work.

Example problem: the BKL (4/4) Lesson learned: “The fact is, any time anybody makes up a new locking mechanism, THEY ALWAYS GET IT WRONG. Don't do it."Take heed. You got it wrong. Admit it. Locking is _hard_. SMP memory ordering is HARD.So leave locking to the pro's. They _also_ got it wrong, but they got it wrong several years ago, and fixed up their sh*t.This is why you use generic locking. ALWAYS.” - Linus Torvalds, 8 Dec 2009

BKL is hard to get rid of Big Kernel Lock was added to kernel - kernel 1.3.54, January 1996 "I suspect we are getting close to the point where we could actually start to remove it.“ - Linus Torvalds, 9 Jun 2001 "At the moment we have the BKL here but this removes the need for it and is a step in the right direction“ - Alan Cox, 29 Sep 2006 "We've had quite a bit of BKL work this merge-window. Maybe we'll even get rid of it one of these days. There are "only" about 600 instances of "lock_kernel()" in the tree right now ;)" - Linus Torvalds, 13 Dec 2009

Reading can be hard, too The most valuable source of Linux information besides the source code is the Linux Kernel Mailing List • Between January 1, 2008 and December 31, 2008 about 150,000 messages were posted • These mails account for 1.2 gigabytes of data • That’s about 410 messages every day (and somebody has to read them...)

Future work(for me, not the kernel people) • Finalize and submit the first article • Find and study more problems • ... and have some fun doing this, because...

Questions? … all work and no play may have some bad consequences: On Tue, 24 Jul 2007, Alan Cox wrote:> > Just one version of Linux ago> The PLL code broke - oh no!> But set the right mode> And fix up the code> Makes the PLL timing sync goAlan, I'm getting a bit worried about you. Linus

Understanding the History of Linux