[06:50] woof [06:52] bark === bigon` is now known as bigon === _bigon is now known as bigon` === bigon` is now known as bigon === bigon is now known as bigon` === bigon` is now known as bigon === bigon is now known as bigon` [18:48] Hi, do you use a second channel when a session is running for people asking questions etc? [18:49] this channel is empty when sessions are not running [18:50] james_w, sometimes, depending on the session [18:58] Kirrus: is there one existing, or is it just set up when needed? === bigon` is now known as bigon [19:19] * slangasek scribbles on the chalkboard === bigon is now known as bigon` === bigon` is now known as bigon [19:44] slangasek: I brought an apple for you [19:44] please tell me you aren't talking about an airbook [19:44] :-) === bigon is now known as bigon` === bigon` is now known as bigon [20:02] hi everyone [20:02] hi [20:02] hehe, finished the text in the last minute :) [20:03] * slangasek sits at attention [20:03] Am I the only student? [20:03] anyone else here for the library packaging session? [20:03] I am [20:04] * warp10 raises an hand :) [20:04] Ah, more students than teachers... [20:04] * db-keen lurks in a seat at the back of the room [20:04] maybe we'll wait five more minutes before we start? [20:04] okay for me [20:04] or just start right away? [20:05] I think it would be more respectful of the time of those who are here to go ahead and start, and if there are latecomers we can recap for them at the end? [20:05] (or point them at a log and invite questions) [20:06] Is there some text we need to look at? [20:06] I agree. [20:06] ok, that's fine, so let's get started :) [20:06] so a big welcome to everyone for joining the motu school session about library packaging [20:06] the session will be split in two parts: [20:07] in the first part, we'll dive deeply into the background of shared objects, and how to find out information about theses [20:07] -s [20:07] and in the second part, we'll go through creating a library package by example [20:07] so let's start with part 1 [20:07] First off, shared objects are useful for two different purposes [20:08] 1) as a plugin mechanism in conjunction with dlopen [20:08] 2) for dynamic libraries, to share code between different projects [20:08] as this is about library packaging, we'll be only looking at 2) in this session [20:09] First off, (ELF) shared objects are a collection of symbols in different sections together with some meta-information [20:09] a symbol denotes a named entity in C (e.g. a function or a variable) [20:09] the section will give us some detail about the symbol [20:10] btw.: if you have questions, just ask ;) [20:10] let's take a look at an example: http://www.potyra.de/library_packaging/example.c [20:10] so please everyone download the file [20:11] everyone got it? [20:11] yep! [20:11] + [20:11] great, so let's compile the file: [20:11] gcc -c example.c -o example.o [20:12] so far, the .o is just a normal object, not a shared object yet... however we can do some stuff with it already [20:12] now run [20:12] nm example.o [20:12] the output are the different symbols defined in example.c [20:14] what are the letters U, T, t... [20:14] for example in line 3, the variable "static_global" is results as symbol "0000000000000000 d static_global" [20:14] mok0: will talk about that in a second ;) [20:14] when you compare the output of nm, together with the variables of example.c, does anyone see anything odd? [20:15] I'll give you a hint: look at the beginning of the function local_function [20:16] now, anyone? [20:16] why local_var isn't listed? [20:16] warp10: excellent [20:16] it's allocated on the call stack [20:17] why does static_local_var has a number in the nm output (here: static_local_var.2268)? [20:17] excactly... some symbols in C (like local_var) don't need to be stored anywhere, because these are located in the stack [20:18] ok, and some variables (like static_local_var) won't result in the same symbols as they are named. This is because the variable is not visible anywhere but in the function, and you can name a variable like this anywhere else [20:18] geser: that's a very good question whose answer I don't remember :) [20:18] and you cannot have two symbols with the same name of course [20:18] geser: could that number be the offset? [20:19] any more questions or shall we go to the funny letters? [20:19] ^^ [20:19] mok0: it's not the offset [20:19] What's the order with which symbols are listed by nm? [20:19] warp10: good question: slangasek? [20:20] plain alphabetical [20:20] slangasek: in order to avoid name clashes between different static local variables with the same name in different functions? [20:20] mok0: I guess it's just a counted number gcc will assign, but I may err on this side [20:20] sistpoty: my number is != gesers [20:20] mok0: it could be the line number after including all headers. after changeing the line with the previous one the number is now 2267 [20:21] siretart: probably, yes [20:21] mok0: he, ah [20:21] static_local_var.1781 [20:21] siretart, geser: suffice to say, it goes away once you've linked the object into a shared library, so it ends up not being very relevant [20:22] ok, then now let's take a look at the letters. [20:22] First off, some are lower case and some are upper case. [20:22] geser: it isn't the line number; if I run the source through gcc -E, I get line 757 [20:22] ok [20:22] The lower case letters denominate local symbols. Noone except this very object file can see these kind of symbols [20:23] lower case == local, Upper case == global [20:23] whereas the upper case letters mean that the symbol is global, and other object files are global... exactly. [20:23] the letter denote the type of the symbol: [20:23] t -> text section (may be mapped readonly), executable [20:23] d -> initialized data (statics, initialized globals) [20:23] (as in variables) [20:24] c -> uninitialized data (uninitialized global variables) [20:24] r -> read only data (const variables), not necessarily read only though. [20:24] (btw, if you later don't remember what these letters map to, they're listed in man nm) [20:24] and then there's an interesting one: u [20:24] this means that the symbol is not defined in the object, but somewhere else [20:25] (undefined) [20:25] to clarify, that means the object *refers* to the symbol but does not define it [20:25] so "stderr" is listed there, as it comes somewhere from the c library, as well as fprintf [20:26] ok, now let's make a shared object from this file: [20:26] gcc -Wl,-z,defs -Wl,-soname,libexample.so.1 -fPIC -shared example.c -o libexample.so [20:27] everyone got it? [20:27] y [20:27] + [20:27] ok, now let's look again at it with nm [20:28] wow... [20:29] (you can ignore anything with an underscore, that's internal stuff) [20:29] nm is like objdump? [20:29] yes [20:29] ok [20:29] I might suggest at this point also running 'nm -D libexample.so', for comparison [20:29] erm... sorry, we need to do another thing I forgot... let's first strip off the debugging symbols [20:30] strip libexample.so [20:30] ... or that :) [20:30] nm: libexample.so: no symbols [20:30] hehe [20:31] that's easy :-) [20:31] ok, to look at the dynamic symbols, you'll now want nm -D as slangasek suggested [20:32] right - since we strip binaries by default, 'nm foo.so' in the wild is going to give you empty output [20:32] the local symbols aren't listed anymore? [20:32] what you'll see is that there are now (apart from some internals) only global symbols left. i.e. everything not defined as "static" [20:32] exactly [20:33] (side note: the static modifier in c means that the resulting symbol will be a local symbol) [20:34] any questions so far? then let's try some other tools on the shared object [20:34] sistpoty: well, global to the functions in that file [20:34] mok0: yes, a local global *g* [20:34] hrh [20:34] geser: since they're local, they're by definition not accessible to other things linking to this code; so aside from debugging symbols, which we've used 'strip' to wipe out, there's no reason to take up space by keeping around names for these variables [20:35] ok, now let's take a look at the meta-information: [20:35] objdump -x libexample.so [20:36] this will give lots of info. The most interesting stuff can be found with objdump -p libexample.so as well [20:37] Who uses anything more than |grep SONAME? ;-) [20:37] one thing that's in there is the SONAME, we'll take a look at it later [20:37] * slangasek raises his hand [20:37] the other thing in there is the NEEDED entry (or entries) [20:38] the SONAME denotes the ABI-Verison of the shared object... [20:38] everything with the same SONAME can be considered to be compatible [20:39] if upstream doesn't make a mistake [20:39] the NEEDED entries denote what libraries this shared object needs. These refer to the SONAME of other libraries [20:39] geser: Never happens ;) [20:39] everything with the same SONAME *will be* considered compatible by ld.so, so it darn well better *be* compatible :) [20:39] geser: we'll just see what happens then ;) [20:40] I guess everyone knows ldd, let's try that one [20:40] ldd libexample.so [20:40] * bddebian doesn't know anything [20:41] i'm late again :( [20:41] ldd will resolve any shared libraries a binary or a shared object needs [20:42] is linux-vdso.so.1 provided by the kernel? [20:42] so, what's the difference between the NEEDED entries and the output of ldd apart from the pathname? [20:42] geser: yes, linux-vdso is a bit of Magic [20:43] ldd output shows three entries, not just one [20:43] ldd resolves them recursively [20:43] * bddebian thinks geser is too smart for this class :) [20:43] exactly what geser said :) [20:43] bddebian: :P [20:43] so you should keep these in mind when dealing with shared objects, as you might want to look at the *direct* dependencies as well [20:44] ldd resolves recursively? [20:44] james_w: yes [20:44] yes; for a real world-example, have a look at ldd /usr/lib/libgnome-2.so.0, vs. objdump -p /usr/lib/libgnome-2.so.0 | grep NEEDED [20:45] so we've looked at the output of nm -D on this object, but nm -D has one weakness as a tool which is not at all obvious if you don't already know what's missing [20:46] have a look at the output of nm -D libexample.so again [20:46] now, compare it to the output of readelf -s libexample.so [20:46] who can spot the difference in the names? [20:47] The stuff from @@glibc? [20:47] you mean the versioned symbols? [20:47] yep - nm -D doesn't show symbol versions [20:47] cool [20:48] so if you want to get at those, you'll want to use objdump -T or readelf -s [20:48] what does the "(2)" in "6: 0000000000000000 144 FUNC GLOBAL DEFAULT UND fprintf@GLIBC_2.2.5 (2)" mean? [20:48] symbol versions are sometimes important, so you'll want to keep these tools in mind -- sometimes, an app/lib will not know the symbol version it's supposed to use, and that indicates a bug! [20:49] geser: I'm not sure, sorry [20:50] I'd probably have to look at the readelf source to figure that out, the fields aren't all well-explained in the documentation [20:50] we can write a dependency resolver from these tools [20:50] mok0: indeed, dpkg-shlibdeps uses these tools [20:51] ok, any more questions so far? [20:51] When writing a program, that uses a library, we want to make sure that the API of that library is stable. [20:52] If we want to distribute this program, we also want a stable ABI. [20:52] stable API means that my program compiles against the library and a newer version of it. [20:52] stable ABI means that the symbols stay the same [20:53] what if there are new symbols? [20:53] mok0: of course... I just wanted to correct myself [20:53] stable ABI means that there won't get any symbols removed, and their meaning is still the same [20:54] sistpoty: if you have a stable API it means if you compiled against it you will be able to link against it? [20:54] james_w: no, that would be stable ABI [20:54] james_w: it means you can compile against it again [20:54] james_w: you can also link against an unstable API/ABI but you need to rebuild everytime the API/ABI changes [20:54] (w.o. changing your code) [20:54] sistpoty: sorry, that's what I meant. [20:55] If you write code that corrects a bug, but the interface remains the same? [20:55] mok0: then the ABI and API is stable [20:56] (most likely) [20:56] ... but I would still want that programs exclusively use the fixed version [20:56] but let's first have some fun... as some newby upstream started to produce a library... [20:56] mok0: if I'm correct you can change the implementation as long as function name, arguments and return type stay the same [20:57] mok0: you can do this (as upstream) only, if the library have some version string or date encoded [20:57] ok [20:57] mok0: or by advetising it on your homepage *g* [20:57] you can bump the minor version number [20:58] for the library: yes (I'll come to this part yet) [20:58] how do you tell if a library has a stable API or ABI is it just by watching the version numbers of that library? [20:58] chiptoxic: I'd like to postpone this question, as I first want to show you an example [20:59] ok [20:59] so please everyone get [20:59] if you've got it, extract it and compile it with [20:59] make [21:00] then install the result (as root) with [21:00] make install [21:00] (no worries, it will only copy files to the /usr/local/ namespace, and make uninstall removes it again) [21:01] everyone got so far? (then me types the commands as well) [21:02] make PREFIX=. install [21:02] does it locally [21:03] then you might have trouble upgrading the library [21:04] (or building an application against it, which we'll do as soon as everyone is finished) [21:04] * warp10 is done [21:04] ok, next step: of course we want s.th. that uses the library, so get [21:05] extract that somewhere else and also build the binary with "make" [21:06] if you've done that, you should have a cool binary hello_prog, which will print hello. you can even tell it how often it should print hello with the first argument [21:06] ./hello_prog 10 [21:07] everyone got so far yet? [21:07] error while loading shared libraries: libmyhello.so.1: cannot open shared object file: No such file or directory [21:07] same here [21:07] mok0: sudo ldconfig [21:07] and here too :) [21:07] heh, did any of the people having this problem install to /usr/local? [21:08] geser: just doing what the teachers tell me .-) [21:08] or did you all install to a different prefix? [21:08] hehe [21:08] works now after ldconfig [21:08] of course I forgot to mention ldconfig [21:08] mine works and i just did what you said and ran ldconfig [21:08] slangasek: /usr/local for me [21:08] ah, ok [21:08] (still got it installed from testing *g*) [21:09] Yay, it says hello to me 10 times [21:09] you have to have /usr/local in your /etc/ld.so.conf of course [21:09] sistpoty: "eventually run ldconfig now." [21:09] kdub: in /etc/ld.so.conf.d/libc.conf actually... :) [21:09] geser: hehe, what a nice upstream *g* [21:10] yep [21:10] now if it prints hello for everyone, let's remove the *library* again with [21:10] make uninstall (run as root) [21:11] now try again to run the hello_prog. [21:12] any results yet? [21:12] ./hello_prog: error while loading shared libraries: libmyhello.so.1: cannot open shared object file: No such file or directory [21:12] boom [21:13] great, so (I guess what you expected) it won't work w.o. the library [21:13] now upstream has released a new version of the library... I guess you all want hot new stuff? [21:13] then get it at http://www.potyra.de/library_packaging/libmyhello-1.0.2.tar.gz [21:13] we'll be compiling it again: [21:13] make [21:14] and install it again (run as root) [21:14] make install [21:14] If you install the library again, it works once more :-) [21:14] hehe [21:14] I don't think so :) [21:15] ./hello_prog: symbol lookup error: ./hello_prog: undefined symbol: print_hello [21:15] (oh, if you haven't removed installed the new library, you could also try ldd ./hello_prog) [21:16] so what happened? [21:16] now let's try to make use of the knowledge we gathered so far [21:16] the application attempts to use the new library [21:17] mok0: exactly [21:17] let's take a look at the symbols of both shared objects (old library vs. new library) [21:17] => ABI has changed [21:17] excellent! [21:18] in the old library, we had the function: T print_hello [21:18] which upstream obviously renamed to just: T hello [21:19] when you start the program, one of the first steps is that undefined symbols will get resolved [21:19] * bddebian curses upstream [21:19] hehe [21:20] yeah, I would think twice about packaging any libraries from this upstream ;) [21:20] which is done recursively for each shared object that is used [21:20] * sistpoty removes copyright notices *g* [21:20] This is why it is wrong to use the program version number for library versions [21:20] heh [21:20] which a lot of upstream programmers don't understand [21:21] when the symbol lookup can't resolve one symbol.. well you just saw what happens. [21:21] ok, what we've seen now is an incompatibility on symbol level. [21:22] I hope with the tools you've learned so far, you can easily track those down [21:22] -> any removed symbol in a shared library is a problem, *if* the SONAME stays the same [21:23] so let's change it [21:24] mok0: feel free to change it, I guess you can figure this in the Makefile [21:24] I set major to 2 [21:25] Wouldn't this be an API change, since print_hello is not defined anymore in hello.h? [21:25] I wouldn't expect this to compile anymore? [21:25] albert23: exactly: this is an API change as well [21:26] ok, but let's get back to the ABI... [21:27] symbols however cannot represent the signature of functions (i.e. number of parameters, type of parameters) [21:27] this is part of the ABI as well [21:27] if we want to look if this changed, we'll need to dig a little bit deeper [21:27] Now the program works again, because it finds the first version of the library [21:27] (I changed MAJOR to 2) [21:28] mok0: hehe, so you didn't unstall the old library? [21:28] im not a programmer but is there a way you could put an alias in the code say print_hello is really hello so you dont break the api? or would that just be messy [21:28] sistpoty: I still have 1.0.1 but I removed 1.0.2 [21:28] The new one is 2.0.1 [21:29] chiptoxic: you could for example abuse the preprocessor: "define print_hello hello" [21:29] My experiment shows you can have several versions of the library present [21:29] er... #define print_hello hello [21:29] ah cool, thanks [21:29] ok, back to the topic... [21:30] you can look at the debug information with gdb, so let's run [21:30] gdb libmyhello.so [21:30] mok0: exactly, and this is a key reason why we always use the full soname for the names of our library packages [21:30] mok0: and now try the same on windows :) [21:30] inside gdb, you can look at the signatures of functions with [21:30] info functions [21:31] slangasek: you mean the major soname ? [21:31] mok0: the "soname" is the value returned by objdump -p $lib | grep SONAME [21:31] slangasek: ok [21:31] and set (hopefully) using the -Wl,-soname option at build time [21:32] mok0: the Makefile just puts $(MAJOR) and a name in the soname, but the real soname is in SONAME [21:32] lintian knows the standard way to map sonames to package names, and warns you if your package is named differently. but of course, we have to take care that upstream is handling their ABI right in the first place! [21:32] inside gdb, you can also display the types of variables with [21:32] info variables === \sh is now known as \sh_away [21:33] (which is uninteresting for libmyhello.so though, since it doesn't define a global variable by itself) [21:33] everyone looked at the output of "info functions"? [21:34] just one: hello(void) [21:34] you should see s.th. like "void hello(void);" or "void print_hello(void);" there (depending which version you chose) [21:34] yes, so upstream is lazy as well *g* [21:35] * bddebian files upstream bug [21:35] now let's strip the shared object (which wasn't done during the build process): [21:35] strip libmyhello.so [21:35] (outside gdb again) [21:36] and then call gdb libmyhello.so again and look at the functions again [21:36] :-( [21:36] as you can see, these are all gone. [21:36] mok0: install the -dbgsym package of it :) [21:37] heh [21:37] that's because these were part of the debug symbols, which have been stripped away [21:37] ok, so if we want to find out about the ABI-part of the signatures, it's a little bit tougher, but still doable [21:39] if a signature is changed (number of arguments and type of a function), it can lead to different results. Most probably it will segfault, since the library and the application no longer agree on what data they exchange) [21:39] an exception to this rule is C++ [21:40] in C++ the signature is encoded in the name of a symbol [21:40] so if you change an argument, add an argument or remove one, you get a different symbol name [21:41] (side note: you can use nm -C to demangle these names back to a human readable signature) [21:41] or c++filt [21:41] any questions so far? [21:42] chiptoxic: did that answer your question from some time ago? [21:42] other side note, checking for ABI changes of this kind by hand is hard and tedious; there is a tool in universe that tries to automate this checking, called icheck [21:42] yea, thanks [21:42] slangasek: icheck, what package? [21:43] slangasek: ah, nice :) [21:43] mok0: icheck [21:43] if you're maintaining a library and want to be careful because your upstream isn't, icheck will let you generate a "manifest" from the old and new -dev packages, and compare them to see if the signatures have changed [21:43] * chiptoxic cant believe how much im learning :) [21:43] slangasek: how does that compare to check-symbols from the ubuntu-dev-tools package? [21:43] * mok0 install icheck [21:43] james_w: icheck examines headers, not symbols [21:44] james_w: so it's specifically for the case where you're trying to detect the kinds of ABI changes that /don't/ show up in the symbol table [21:44] * bddebian can't keep up being at work :'-( [21:45] (doh, check-symbols uses nm -D? Baaad, what did we learn was the weakness of nm -D? :-) [21:45] slangasek: since you mentioned ABI changes not showing up in the symbol table, want to say a few words to these? [21:46] probably, let me see where I put those words [21:47] right - does everyone here understand why changing the arguments or return type of a function is an ABI change? [21:47] yes [21:47] (if everyone knows, there's no sense in me spending time explaining it - but if you don't know, please speak up :) [21:48] y [21:48] Well renaming a global variable would show up [21:49] two people know who care, everyone else is asleep or doesn't want to talk to me ;) [21:49] sistpoty: back to you [21:49] :) [21:49] I just tried icheck on the 2 library versions, [21:50] but I couldn't get it to work [21:50] ok [21:50] ... "Failed to parse libmyhello.so [21:50] mok0: haven't used it myself yet [21:51] sistpoty: :-) [21:51] what we should have learned so far is that if the ABI breaks, you'll need to bump the SONAME [21:51] mok0: you point it at the headers, not at the library [21:51] Ah! [21:51] however there is yet another case, we didn't consider [21:51] that's the whole point - the library doesn't have the info you need about the function sigs :) [21:52] upstream might put additional functionality in the library [21:52] and keep the ABI downwards compatible [21:53] in that case, the SONAME would (and should) stay the same, because no program ceases to work with the newer library [21:53] however this has one caveat: people who use this library might make use of the new features [21:53] ... but new programs that use the new functionality? [21:54] right, so these programs wouldn't work with the old library but only with the new one (even though the SONAME is the same) [21:54] for this, there the SONAME is not sufficient alone. [21:55] slangasek: told me that historically the minor version was incremented for this [21:55] -: [21:55] * mok0 also read that [21:55] but that won't work if you copy binaries from one system to another (as you don't know about the libraries there) [21:56] ok, this pretty much was part 1 -- shared objects [21:56] any questions so far? [21:57] Well, what is the answer to the last problem? [21:57] mok0: we'll see that in part 2 :) [21:57] Arrghh [21:57] :) [21:57] It stops at the most exiting part [21:58] typical of these mini-series :-) [21:58] hehe [21:58] :D [21:58] actually I wanted to continue with part 2 now, unless there are questions of course [21:59] anyone? [21:59] * warp10 has no questions [21:59] ok, now for part 2 the plan was to let you package libmyhello on a walkthrough basis [21:59] however the session went much longer than I expected [22:00] 2 hours [22:00] would you prefer the walkthrough or shall we just look at (one possible) result? [22:00] I'm ok with the latter [22:00] sistpoty: would you like to move that part to another session? [22:00] any choice is fine for me [22:00] * albert23 would like the walkthrough [22:00] that would be an option as well [22:01] what does everyone else think? [22:01] sistpoty: I think a walkthrough would be really valuable, but I'm sure you are tired. [22:02] maybe moving to another session could give the opportunity to go in deeper details than going on now. [22:02] james_w: heh, a little bit, but tomorrow is friday so I don't mind going to work tired *that* much *g* [22:02] +1 [22:03] ok, then let's move this to another session... maybe tomorrow? [22:03] Works for me [22:03] warp10: what about you? [22:03]