=== \sh is now known as \sh_away === bigon is now known as bigon` === Pricey is now known as PriceChild === BaD-Laptop is now known as BaD-AcerLaptop === \sh_away is now known as \sh [08:34] loot,are u there === coolcat is now known as Rhi_evil_happy === coolcat is now known as Rhi_evil_happ1 [09:02] hi everyone [09:03] anyone around for the library packaging session? (repetition of the session from Thursday)? [09:05] hello [09:05] Hi [09:05] hi [09:05] hi [09:06] you want to hear the library packaging session? [09:07] sistpoty: if there is a log from the other one, I am happy to read that, but if this is going ahead in here, I wil idle. [09:07] TheMuso: there is a log available, let me look [09:07] sistpoty: Thanks. [09:08] TheMuso: http://irclogs.ubuntu.com/2008/01/17/%23ubuntu-classroom.html [09:08] *looks happy* [09:08] sistpoty: Thanks a lot. [09:09] TheMuso: and yesterday was part two: http://irclogs.ubuntu.com/2008/01/18/%23ubuntu-classroom.html [09:09] right [09:09] ok, then let's get started, shall we? [09:10] Bookmarked [09:10] this session is made up of two parts [09:11] in part 1, you'll get to know everything about shared objects [09:11] in part 2, we'll package up a sample library and discuss some packaging issues [09:11] let's start with part 1 [09:11] first off, shared objects are useful for two purposes [09:11] 1) they can be used as a mechanism for plugins with dlopen [09:12] 2) to share code between different projects (a library) [09:12] we'll deal with 2) in this session :) [09:12] oh, btw.: if there are any questions just ask them right away [09:13] a (ELF) shared object is a collection of symbols in different section together with some meta information [09:13] a symbol denotes any named entity in C (functions, variables) [09:13] let's get an example and see what it will result in [09:14] [09:14] the section a symbol is gives details about it (e.g. it can be executed, it's read only...) [09:15] everyone got the example? [09:15] yes [09:16] so let's compile it: [09:16] gcc -c example.c -o example.o [09:16] this will just result in an ordinary object (not a shared one, we'll look at this in a moment) [09:16] let's take a look at the symbols in there [09:17] we can use readelf, nm or objdump for it [09:17] let's start with [09:17] nm example.o [09:17] the rightmost column is the name of the symbol... let's compare it with what's in the c-file [09:18] http://paste.ubuntu.com/3667/ [09:19] for example, the global_function from line 26 will turn into the symbol with the same name [09:19] 000000000000006b T global_function [09:19] anything odd you can find of the symbol output? [09:19] if I get less zeros at start that's because I've a different arch or what? :P [09:20] RainCT: most probably *g* [09:20] I get "static_local_var.2268" rather than static_local_var [09:20] persia: excellent [09:21] first off the static_local_var comes from line 15 [09:21] as you cannot have two symbols with the same name in one object [09:21] static local variables need to have their name mangled [09:22] Is that just by policy, rather than by checking to see if there would be a conflict? [09:22] because you can define a static local variable with the same name in two different functions [09:22] persia: not too sure if you could produce (somehow) an object with two equally named symbols [09:23] the linker wouldn't allow this, but maybe it could be circumvented [09:23] however the second symbol would never get looked at [09:23] sistpoty: Sorry to not be clear: I just don't see two different static_local_var, so wondered if the compiler just added index codes, rather than doing any conflict check. [09:23] (you can link two objects with equally named symbols together, but then the second one is discarded) [09:24] persia: the conflict check is getting done by the linker [09:25] OK. So the compiler mangles by policy to avoid conflict issues at link-time? [09:25] persia: and I guess that the compiler will just add the index for every static local [09:25] I believe so [09:25] Thanks for the details :) [09:26] at least that's what I'd do as a compiler... because I already new that's a static local, so just add an index and not needing to check for collisions afterwards :) [09:26] ok, anything else that's odd with the nm output compared to the c-file? [09:26] (line 10 *cough*) [09:27] in line 10, we've got a variable local_var, which won't result in any symbol [09:27] the reason for this is, that it's locally to the function and will get allocated on the stack [09:27] hence there's no need to allocate space for it in the object [09:28] now let's make a shared object from the example [09:28] Was it not optimised away by the compiler? It's only used once. [09:28] persia: not without -O(something) [09:28] Ah. Right. [09:29] gcc -Wl,-z,defs -Wl,-soname,libexample.so.1 -fPIC -shared example.c -o libexample.so [09:29] the -Wl,something are options gcc will pass to the linker [09:30] the first one is only for sanity (allow no unresolvable symbols) [09:30] and the second one defines the soname (we'll come to this bit later on) [09:30] -fPIC will result in position independent code, but I won't go into details here [09:30] and -shared tells gcc to produce a shared object [09:30] let's strip the object [09:31] strip libexample.so [09:31] take a look at it again [09:31] nm libexample.so [09:32] at first sight, there don't seem to be any symbol information stored in there [09:32] let's try again [09:32] nm -D libexample.so [09:32] this will display dynamic symbols [09:33] now compare the output to the one from example.o [09:34] at first, there are a number of symbols starting with an underscore... these are (compiler) internals, so we'll ignore these [09:34] let's take a look at the letters before the symbol name [09:34] as you can see, there are only capital letters left in the shared object [09:35] the capital letter means, that the symbol is a global symbol [09:35] i.e. other object files linked against it can see it [09:35] if it's lower case, it's only visible inside that particular object [09:36] (and hence you can link two objects with symbols that have the same name but are only locally visible together w.o. problems) [09:37] as the symbols in shared objects are meant to be used by different programs/libraries, we're only interested in global symbols [09:37] let's take a look at the letters in more details [09:37] the letters describe in what section the symbol lives in [09:37] t -> text section (may be mapped readonly), executable [09:37] d -> initialized data (statics, initialized globals) [09:37] c -> uninitialized data (uninitialized globals) [09:38] r -> read only data (const variables), not necessarily read only though. [09:38] and then there is a special one: u [09:38] u means it's undefined in this object, and comes from another object [09:39] symbols that example.c use (for example stderr) but that are not declared in there, will get the U [09:40] if you have a binary, that has undefined symbols, these will get resolved by the loader, as soon as the program is executed [09:40] any questions so far? [09:40] What is B? [09:41] ranf: b is uninitialized data [09:42] ranf: so the global extern_var will live there (the initial value is not getting set) [09:42] btw.: you can always look at the manpage of nm to find out about the different letters [09:42] ok, let's take a look at the meta-information [09:43] objdump -x libexample.so [09:43] the interesting parts here are [09:43] NEEDED entries [09:43] and SONAME [09:43] if you prefer less output, you can also look at these with [09:43] objdump -p libexample.so [09:44] the NEEDED entry refer to the SONAME entry of shared objects, that this shared object directly needs [09:45] i.e. all undefined symbols in libexample.so will get resolved to the shared object where these are defined [09:45] and each shared object where such a symbol comes from is listed as NEEDED [09:46] let's compare this with [09:46] ldd libexample.so [09:46] ldd will basically do the same thing, that the loader would do [09:46] How was libexample.so.1 determined? I usually see some sort of definition in upstream build systems. [09:47] persia: it should match the soname [09:48] err, sorry the resulting shared object should match the soname by some rules [09:48] here it was just what we gave to gcc, w.o. any big thoughts how to call it [09:48] sistpoty: Right. objdump -x or -p reports "SONAME libexample.so.1" I just don't understand where the "1" comes from. [09:48] Ah. Right. The compilation command. Sorry. [09:49] persia: that's exactly what we gave to gcc: -Wl,-soname, [09:49] if you look at the output of ldd, you can see that [09:50] 1) it resolves the needed entries to path names of the shared objects in question [09:50] and 2) that there are more entries, not listed in needed [09:50] that's because ldd will do lookups recursively [09:51] so it will first find libc.so.6 and look at what's needed by that one as well [09:51] and list all of these [09:51] let's rather look at a real world example to spot the difference [09:51] ldd /usr/lib/libgnome-2.so.0 [09:51] compared to [09:52] objdump -p /usr/lib/libgnome-2.so.0 | grep NEEDED [09:53] any questions so far? [09:53] ok, now let's also compare [09:54] nm -D /usr/lib/libgnome-2.so.0 [09:54] to [09:54] readelf -s /usr/lib/libgnome-2.so.0 [09:54] * RainCT has to go and will check the log when he's back [09:54] (or you can also compare libmyexample.so) [09:54] cya RainCT [09:55] spot any difference? [09:56] if you use readelf -s, you'll see that some symbols have an @ in their name [09:56] these are versioned symbols [09:56] nm will happily not print out that info [09:57] so when looking at shared objects for library packaging, you should prefer readelf -s to nm -D [09:57] objdump -T libmyexample.so will also display information about versioned symbols [09:58] when we have a library, we're interested to have a stable ABI [09:59] a stable ABI means that there won't get any symbols removed, and that the symbols mean the same thing [09:59] in contrast a stable API means that we can compile a program with one version of the library and can compile it again with another version [09:59] w.o. changing the code of the program [10:00] let's try s.th. out [10:00] http://www.potyra.de/library_packaging/libmyhello-1.0.1.tar.gz [10:01] this is a very simple example library [10:01] extract it, and compile it with [10:01] make [10:01] please also install it (as root) with [10:01] make install [10:01] no worries, it will only install under /usr/local, and comes with make uninstall as well to remove the craft [10:02] no we'll need a program that uses this library as well [10:02] now [10:02] get http://www.potyra.de/library_packaging/hello_prog-1.0.tar.gz [10:02] extract it to a different directory and compile it [10:03] everyone got that so far? [10:04] oh, be sure to run ldconfig after installing the library package, I always forget this [10:04] everyone got a program called hello_prog now? [10:05] anyone still following? *g* [10:05] * persia has a working hello program (after running ldconfig) [10:05] great :) [10:06] now for a test, let's uninstall the library again [10:06] make uninstall (as root [10:06] +) [10:06] let's try to execute hello_prog again [10:07] it won't work, because the shared object is gone for good [10:07] if you try ldd hello_prog, you'll see that it cannot resolve libmyhello.so.1 [10:07] luckily, upstream has made a new library package, let's try that hot new stuff [10:08] http://www.potyra.de/library_packaging/libmyhello-1.0.2.tar.gz [10:08] same procedure as with the old libmyhello: compile, install, run ldconfig [10:09] * persia complains upstream shouldn't distribute compiled binaries in the tarball [10:09] hm? does upstream do that? [10:10] sistpoty: Maybe it's just my upstream checking scripts :( [10:10] persia: probably (or my make dist is wrong) [10:10] ok, let's retry to run hello_prog [10:11] at this point the loader cannot resolve a symbol, and will barf out [10:11] let's take a look: readelf -s hello_prog [10:12] (first run strip, to get rid of the local symbols) [10:12] strip hello_prog [10:12] readelf -s hello_prog [10:12] you can see, that it wants print_hello [10:12] let's take a look at what the new library provides [10:13] strip the library (to get rid of the local symbols) [10:13] and [10:13] readelf -s libmyhello.so [10:14] there is a function hello, but no function print_hello [10:14] compare that to what's in the old libmyhello.so [10:14] so upstream obviously removed print_hello and inserted hello [10:15] this means, that the change is not ABI compatible (and also not API compatible, because you cannot compile hello_prog against the new versoin) [10:15] so obviously if a symbol gets removed, it means that the ABI breaks [10:15] let's come back to the SONAME [10:15] having the same SONAME means that the ABI is compatible [10:16] so you can exchange any library with the same SONAME and the program should still work [10:16] if a library isn't ABI compatible any longer, the SONAME must get changed to indicate this [10:17] if the SONAME gets changed, so does the names of the library [10:17] so you can have two versions of one library installed, that have a different SONAME [10:17] any questions so far? [10:18] so let's get back to the ABI again... [10:18] having the same symbols (or only new symbols) in a newer version is only part of the ABI definition [10:19] Is the dual-installation not affected by the symlinks? [10:19] persia: no [10:19] maybe I should go into details about the symlinks [10:20] for one, we have the shared object itself: libmyhello.so.1.0.2 [10:20] I'd like to know more, but perhaps in section 2 about packaging. It seems to me the symlink points to the most recently installed, which is non-deteministic. [10:20] well, I'll just explain that now ;) [10:21] the first symlink is libmyhello.so.1 [10:21] this one is actually always the SONAME mangled to some rules [10:22] err.. it *is* the SONAME [10:22] sorry [10:22] if this symlink isn't there, ldconfig would create it [10:23] all binaries, that want a specific shared library are resolved by this symlink [10:23] because that's what's in the NEEDED entry of the binary [10:23] the loader will do that for us, and (looking at ldd output) will also follow that symlink to the "real" shared object [10:24] the other symlink is libmyhello.so [10:24] this symlink is there, if you want to link a binary against the shared object [10:24] gcc -lmyhello [10:25] -l will get transformed to lib.so [10:25] and that's what the linker will search for [10:25] Ah. That explains why the bare symlink is usually in the -dev package. Thanks. [10:25] so you'll only need this symlink when compiling/linking s.th. against a shared object [10:25] exactly [10:26] any binaries won't need/see it. and that's the only file with the same name [10:26] ok, let's get back to the ABI [10:27] the ABI also means that the symbols still mean the same thing [10:27] while finding out, if any symbols got removed should be an easy task now, this gets a little bit trickier [10:27] hi [10:27] the meaning of the symbol is 1) is it a function/variable/what section is it in [10:27] hi Rhi_evil_happ1 === Rhi_evil_happ1 is now known as Rhi_evil_happy [10:28] but also for functions, what signature (number of arguments, type of arguments) need to get passed [10:28] hi again [10:29] if you change the signature of a function of a library, this means that a program linked against it wouldn't agree with the library any longer, about the data that it passes [10:29] one way to look at signatures is via the debug information [10:30] so (in case you stripped the library), recompile it again w.o. stripping it (make clean && make) [10:30] for debug information, gdb is our tool of choice [10:30] gdb libmyhello.so [10:31] you can look at the function definitions with [10:31] info functions [10:31] void hello(void); [10:31] ? [10:31] Rhi_evil_happy: maybe you want to read backlog? [10:32] Just to confirm. this would provide useful information if the function weren't void, right? [10:32] persia: yes [10:32] s/useful/visibly useful/ [10:32] feel free to modify the functions to your liking and experiment ;) [10:32] for variables it's [10:32] info variables [10:32] (however there is no direct variable defined in there) [10:33] interesting enough stdin/stdout show up there... I guess that's because these are declared in a header and will get debug symbols in the library... [10:34] so to be sure, you'd need to filter out any info from there that's not marked as a symbol [10:34] or rather that's not defined in a symbol (nm -D/readelf -s/objdump -T) [10:34] now, let's strip the library again and see what info gdb will spit out [10:35] stirp libmyhello.so [10:35] gdb libmyhello.so [10:35] info functions [10:35] s/stirp/strip/ [10:36] now, the info is gone... since all binaries/shared objects are stripped, this cannot get applied to a package in the archive [10:36] or at least you'd need the -dbgsym package for it [10:36] luckily there exists one tool, that may help there: icheck [10:37] afaik, you can use it to create a manifest file with symbols (and their) meaning of a package [10:37] I haven't tried it out myself yet though [10:37] (package icheck) [10:38] there also exists check_symbols in the ubuntu-devscripts package, however we found out that it currently uses nm -D [10:38] and thus won't know anything about versioned symbol dependencies [10:38] -> better avoid it until its fixed [10:38] any questions so far? [10:39] ok, small side note: c++ libraries [10:40] since you can overload method in c++, (give two functions the same name but different signatures), [10:40] these cannot get stored with the name as symbol name [10:40] because you'd end up with two symbols with the same name [10:41] hence there is some name mangling done by g++, that encodes the signature in the symbol name itself [10:41] let's take an example [10:41] nm -D /usr/lib/libstdc++.so.6 | less [10:42] (there are lot's of symbols with funny names in there) [10:42] you can get some human readable output of this with the -C option of nm [10:42] nm -C -D /usr/lib/libstdc++.so.6 | less [10:43] or c++filt [10:43] a nice side effect of this is, that a change of the signature will result in a differently named symbol [10:44] and hence in a removed symbol, which you can easily detect looking only at symbols [10:44] another side note: [10:44] Is this use of guard symbols why C++ gets so hopelessly confused with insufficiently explicit casts to objects passed to overloaded functions? [10:45] what do you mean with guard symbols? [10:45] however it gets confused due to overload resolution [10:45] that means the compile needs to find out which method to call [10:46] The output of `nm -CD /usr/lib/libstdc++.so.6 | less` reported "guard ..." in place of _ZZZ09u592. My terminology error. [10:46] I'm not too sure what the guard thingy means [10:47] however if the type for an argument exactly matches, this method is chosen by the compiler (i.e. uint64_t as parameter, uint64_t as argument) [10:48] if an implicit typecast is involved (e.g. uint32_t as parameter, uint64_t as argument), this is considered a worse case [10:48] Thanks. Somewhat unrelated then :) [10:48] iirc, only the number of typecasts involved is getting counted, and the method is chosen for this [10:49] persia: maybe it's also about NULL as argument (which's type can be any pointer type) ;) [10:49] so void do_stuff(X *arg); and void do_stuff(Y *arg); would be ambiguous when called as do_stuff(NULL); [10:50] but let's get back to shared objects [10:50] sistpoty: I've only encountered it when porting applications to updated libraries. Likely off-topic :) [10:50] persia: heh, I encountered it yesterday at work, coding c++ *g* [10:51] ok, so the SONAME means to be ABI compatible [10:51] and by definition, if we only add new features to a library, we don't need to change the SONAME [10:51] this leaves one small problem: what if we've added cool new features not breaking the ABI [10:52] and a program makes use of these new features [10:52] someone could still have the old library installed and install the program [10:52] then this program wouldn't work any longer, because it needs symbols not yet defined in thre [10:52] there [10:52] on the basis of the SONAME, there is no solution to this [10:53] historically on unix systems, this was handled by the minor version of a shared library as slangasek told me [10:53] however this of course won't work if you move binaries from one system to another (and that's what packaging is about) [10:54] we'll see the solution to this in part 2 [10:54] ok, any questions left? [10:55] if there are none, I'm through with part 1 :) [10:55] If the minor number is no longer used to track symbol introduction, what is it used to indicate? [10:55] heh, no idea actually [10:56] maybe it's still used for this [10:56] Is it just meaningless on linux/glibc systems then? [10:56] yes [10:56] i.e. I'm not sure if it's meaningless, but debian/ubuntu systems could live w.o. it ;)( [10:57] Actually, only the runtime loader matters. If this doesn't use it, it is meaningless, even if there is historic semantic value. [10:58] right [10:58] how about a short break before starting part 2? [11:00] ok, I'm just making a short coffee break then :) [11:14] re [11:15] anyone around for part 2 of the library packaging session? [11:16] persia: interested in part 2? [11:16] (if not, I'll just point to the logs... less work for me then *g*) [11:17] Sure. [11:18] * persia reviews the logs again anyway, to forestall some questions [11:18] persia: if you're the only one interested, I guess I don't need to go through packaging by example and just skip ahead to questions, ok? [11:19] sistpoty: OK. Hold on a bit while I catch up then :) [11:19] sure, just ping me once you're through ;) [11:21] sistpoty: question time :) [11:21] :) [11:22] First, are there any objective criteria about pushing a transition vs. oldlibs support? Is this just a matter of how much effort the transition is, and based on what is expected to make the release? [11:23] hm... I can only say what slangasek wrote yesterday, that it's /usually/ better to transition [11:23] I guess one example would be the libdb* stuff, which is an exception [11:23] because it's a database library and the database format changed iirc, so it's very hard to transition this one [11:24] and would result in a very unsmooth upgrade path [11:24] or you'll also like to not get rid of libc* too soon, as many binary only applications still use the old one [11:24] Makes sense. libdb seems a prominent exception, and we've been having fun with WX since feisty. I'm guessing that it depends on 1) if upstreams are paying attention, 2) bad upgrade paths, 3) truly significant porting effort (that won't complete in several months), and similar release-affecting issues. [11:25] sounds sane, yes [11:26] Next: do you understand the controversy over binary packages named libfoo1 vs. libfoo1.0.3? If so, why one over the other (or do you have a pointer to a previous debate)? [11:27] hm... [11:28] to my understanding, the binary package name should have a unique name in comparison to the soname [11:28] so libfoo1 would be useful for libfoo.so.1 [11:28] while libfoo1.0.3 would be useful for libfoo-1.0.3 (soname) [11:29] Hmm. That would make more sense if the minor version weren't considered meaningless :( [11:29] well, the minor version isn't part of the SONAME, otherwise it's no minor version ;) [11:29] Does maybe the versioned symbols features allow parallel installation of libfoo-1.0.3 and libfoo-1.0.4? [11:30] no, the only warranty for parallel installation is a different SONAME afaik [11:30] * persia looks at libmyhello Makefiles again [11:31] versioned symbol come in at a different place [11:31] let's say, you have libA and libB, which both depend on libC [11:31] Ah. So libfoo1.0.3 would provide libfoo1.0.3.so.(whatever), rather than libfoo1.so.1.0.3 ? [11:31] persia: exactly [11:32] btw.: the library packging guide has nice examples for this [11:32] sistpoty: re: versioned symbols. Right. Poor terminology again (and more interesting for package D depending on libA and libB). [11:32] right [11:33] if you now change libA to require a newer, incompatible version of libC [11:33] D will draw in multiple versions of libC [11:33] Next question: if an upstream tarball produces 17 libraries, why would one choose e.g. 5 (10) binary packages, rather than 17 (34)? [11:34] good question... the only rationale is if these come from the same source and one ABI incompatibility of one shared object will *always* result in ABI incompatibilities of the other shared objects [11:35] if you know that this is the case, you can collapse these into one library package (because you'll never end up with only one SONAME getting bumped) [11:35] So if there is a stack of binary libraries generated by the same source, and the core library (upon which the others depend) is generally updated for each release, and applications usually use several of the client libraries, one package might be correct, but upstream should be hit with a stick? [11:37] well... kind of [11:37] it might be correct though... let me think of an example [11:37] kdelibs is perhaps an interesting example [11:37] if the core library defines some interface types, that are used in the client libraries [11:38] you'll automatically get ABI incompatibilities if the core library changes [11:38] Err.. No. That generates lots of binaries. [11:38] becaues the signatures will differ [11:39] hm... not too sure about kdelibs *g* [11:39] Won't that generally be taken care of by the simultaneous recompile of the single source, regardless of how many binary packages are involved? [11:40] yes, it will [11:41] OK. I'll put the package split question back on the stack of things to have an opinion about rather than things with an answer then :) [11:41] Next question, which may be off-topic: [11:41] however if you put (independent) shared objects into a single binary, it means that every program depending on any to rebuild as well [11:41] so a good rationale would also be if it's possible to use only one single shared object or if you definitely want all of these [11:42] sistpoty: Ah. Right. Splitting the binaries and being careful about versioning would decrease the amount of NBS work if clients typically used subsets of the set of libraries exported by the source. Thanks. [11:43] and unmet dependencies [11:44] When packaging an application that has internal libraries that are tightly linked to the application itself (and not expected to be used externally (e.g. btanks)), is there a point to exporting the library, or would just putting it in the /usr/lib/ hierarchy (for ldconfig) be sufficient? [11:44] (I consider library related unmet deps to be unfinished NBS, but that's just nomenclature) [11:45] blatantly: if it comes with headers and you can compile s.th. against the the library in question, it should be packaged as a library [11:45] otherwise (e.g. headers alone not sufficient to build against), somewhere under /usr/lib/??? sounds sane [11:46] OK. So headers are the useful defining characteristic. I'll be reusing that argument next time I am trying to defend a package split. [11:46] kind of... usually if you have a shared object only for internal use, there is no much sense for the shared object in the first place ;) [11:46] (why not link statically then) [11:47] of course apart from plugins... which is a different use case [11:47]