/srv/irclogs.ubuntu.com/2016/01/15/#juju-dev.txt

anastasiamacmenn0: ping00:19
anastasiamactanzanite was planning to do the renaming of all env to model once controller-rename branch gets into master00:20
menn0anastasiamac: pong00:20
menn0anastasiamac: how were you going to handle things like DB collections?00:20
menn0anastasiamac: and what about API names?00:21
anastasiamacmenn0: we have not gotten to the point of actually discussing it yet as controller-rename branch is still pending in limbo somewhere...00:21
menn0anastasiamac: ok np00:21
anastasiamacmenn0: i suspect that we will have a brief next week where we'll discuss the approach00:21
anastasiamacmenn0: i'd love to get anyone who is interested involved00:22
anastasiamac(u sound like one)00:22
anastasiamacinvolved in discussion that is :D00:22
menn0anastasiamac: would you mind replying to my email then... if there's going to be a major rename effort so that "environment" goes away completely then I'm fine with using "model" everywhere00:23
menn0anastasiamac: I think the rename is going to be hard to get right00:23
anastasiamacmenn0: k. and I agree.... hence i think we should brain-storm before we dive into actual rename \o/00:24
menn0anastasiamac: but if anyone can do it it's your squad :)00:24
anastasiamacwot?!00:24
menn0anastasiamac: I'm happy to participate in the discussion and planning00:24
anastasiamaci think with onyx's experience and the lessons learnt from controller-rename, ur squad is very well positioned to do it ;D00:24
anastasiamacawesome!00:24
menn0anastasiamac: onyx has been banned from working on anything but model migrations00:25
menn0anastasiamac: but since this affects model migrations I think we can at least be involved in the discussion00:25
anastasiamacmenn0: k. xross-squad pairing sounds awesome :D00:26
menn0sure :)00:26
menn0probably not going to happen though00:26
anastasiamac:( i can dream00:26
anastasiamaci have a dream?...00:26
menn0:)00:28
davecheneycherylj: chasing this https://bugs.launchpad.net/juju-core/+bug/153439400:53
mupBug #1534394: trusty/i386 test failure <juju-core:New> <https://launchpad.net/bugs/1534394>00:53
cheryljdavecheney: I think that waigani already fixed that bug.01:00
cheryljbug 153277701:01
mupBug #1532777: undertakerSuite.TestEnvironConfig fails <blocker> <ci> <regression> <unit-tests> <juju-core:Fix Committed by waigani> <https://launchpad.net/bugs/1532777>01:01
cheryljdavecheney: the current things we're blocked on may be CI test issues.  So at this point, I don't have specific bugs for you.01:01
cheryljwe will need some help with wily / xenial unit tests on 1.25 as we're extending its support for a while01:01
cheryljbut I've been focused on 2.0-alpha1 right now and don't have a list of those bugs yet01:02
cheryljdavecheney: so I don't need you right now.01:03
cheryljwas that rambling enough?01:03
mupBug #1534394 opened: trusty/i386 test failure <juju-core:New> <https://launchpad.net/bugs/1534394>01:09
davecheneycherylj: noted01:18
davecheneyi'll move on to something else01:18
ericsnowkatco: crickets01:51
mupBug #1534394 changed: trusty/i386 test failure <juju-core:New> <https://launchpad.net/bugs/1534394>05:07
mupBug #1534480 opened: LXD: bootstrap error: no registered provider <juju-core:New> <https://launchpad.net/bugs/1534480>07:52
mupBug #1534511 opened: Support for multiple apt sources and apt keys in isolated environments <cpe> <stakeholder> <juju-core:New> <https://launchpad.net/bugs/1534511>09:19
frobwaredimitern, voidspace, dooferlad: will be 10-15 mins late for standup.09:46
dooferladfrobware: I am happy to start late09:46
dooferladfrobware: ping us when you are ready?09:46
dimiternsure09:54
frobwaredimitern, voidspace, dooferlad: ready10:13
* dooferlad joins hangout10:14
voidspaceomw10:18
voidspacedimitern: frobware: dooferlad: still need a review by the way: http://reviews.vapour.ws/r/3534/10:19
mupBug #1534480 changed: LXD: bootstrap error: no registered provider <juju-core:Invalid> <https://launchpad.net/bugs/1534480>10:31
voidspacefrobware: dimitern: dooferlad: so in the merge provider/maas/environ.go is fairly screwed11:07
voidspacethis is becauase a bunch of code was broken out into separate files on maas-spaces11:08
voidspacebut fixes applied to master (for default gateway stuff)11:08
mgzbless on controller-rename btw11:17
voidspacefrobware: dimitern: dooferlad: coffee11:51
voidspacedimitern: frobware: dooferlad: still need a review by the way: http://reviews.vapour.ws/r/3534/12:50
frobwarevoidspace, ack12:50
voidspacedimitern: frobware: dooferlad: my merge branch now compiles, except for the tests12:50
voidspacewhich are still pretty screwed12:50
frobwarevoidspace, nice12:51
voidspaceas soon as they compile I can see what's broken12:51
voidspaceI'm worried that fixes that landed on master in code that was extracted into separate files on maas-spaces might get lost12:51
voidspacethat's the biggest danger12:51
voidspace*hopefully* the tests will pick that up12:51
voidspacebut tests got moved too12:51
voidspaceso it'll need a good review to make sure we haven't lost fixes in the merge12:51
cheryljcan I get a review?  http://reviews.vapour.ws/r/3546/13:31
cheryljIt's a quick one13:31
mupBug #1521777 changed: Allow for upgrades to 2.0 <juju-core:Fix Released by cherylj> <juju-core 1.25:Fix Released by cherylj> <https://launchpad.net/bugs/1521777>13:38
dimiternvoidspace, you've got a review btw13:55
perrito666lazyPower: ping14:17
voidspacedimitern: cool - thanks14:17
voidspacedimitern: ah, I didn't know about the "if !c.Check(...) {continue}" trick14:18
voidspacedimitern: that's why I wasn't using check in that loop14:19
voidspacethanks14:19
dimiternvoidspace, :) yeah - it's useful14:20
frobwaredimitern, voidspace, dooferlad: let's use https://plus.google.com/hangouts/_/canonical.com/juju-sapphire14:30
voidspacefrobware: really?14:30
voidspaceI'm already in the one you linked to14:30
voidspaceah well.14:30
voidspaceomw14:31
frobwarevoidspace, ah, sorry. I tried to change it to that but the juju-sapphire URL but it obviously didn't work.14:31
dimiternfrobware, omw, 2m14:32
lazyPowerperrito666 pong14:40
mgzcherylj: lgtm14:45
cheryljthanks, mgz14:45
natefinchchery, sorry I missed your emssage last night. Yes, that's what I'm doing now.  Trying to repro with the steps in the bug.  I reviewed fwereade's patch, which looks pretty sane... but I haven't been able to reproduce the bug yet, so it's hard for me to tell if the fix will fix it.14:54
natefinchcherylj: ^14:54
cheryljnatefinch: thanks for the update!14:55
mgzoh, I thought we were all calling her 'chery' now...14:57
mupBug # opened: 1534610, 1534619, 1534620, 153462315:23
mupBug #1534626 opened: provisionerSuite.SetUpTest fails because connection reset <ci> <intermittent-failure> <reliability> <juju-core:Triaged> <https://launchpad.net/bugs/1534626>15:29
mupBug #1534627 opened: Destroyed models still show up in list-models <juju-core:Triaged> <https://launchpad.net/bugs/1534627>15:29
perrito666lazyPower: sorry I solved itt myself, thank you anyway15:29
lazyPowerah bueno perrito666  :)15:30
perrito666that is pretty much like throwing a wrench into my brain gears15:31
perrito666I was considering for a moment what channel was this15:31
mupBug #1534626 changed: provisionerSuite.SetUpTest fails because connection reset <ci> <intermittent-failure> <reliability> <juju-core:Triaged> <https://launchpad.net/bugs/1534626>15:32
mupBug #1534627 changed: Destroyed models still show up in list-models <juju-core:Triaged> <https://launchpad.net/bugs/1534627>15:32
mupBug #1534626 opened: provisionerSuite.SetUpTest fails because connection reset <ci> <intermittent-failure> <reliability> <juju-core:Triaged> <https://launchpad.net/bugs/1534626>15:38
mupBug #1534627 opened: Destroyed models still show up in list-models <juju-core:Triaged> <https://launchpad.net/bugs/1534627>15:38
mupBug #1534626 changed: provisionerSuite.SetUpTest fails because connection reset <ci> <intermittent-failure> <reliability> <juju-core:Triaged> <https://launchpad.net/bugs/1534626>15:41
mupBug #1534627 changed: Destroyed models still show up in list-models <juju-core:Triaged> <https://launchpad.net/bugs/1534627>15:41
mupBug #1534626 opened: provisionerSuite.SetUpTest fails because connection reset <ci> <intermittent-failure> <reliability> <juju-core:Triaged> <https://launchpad.net/bugs/1534626>15:44
mupBug #1534627 opened: Destroyed models still show up in list-models <juju-core:Triaged> <https://launchpad.net/bugs/1534627>15:44
mupBug #1534626 changed: provisionerSuite.SetUpTest fails because connection reset <ci> <intermittent-failure> <reliability> <juju-core:Triaged> <https://launchpad.net/bugs/1534626>15:47
mupBug #1534627 changed: Destroyed models still show up in list-models <juju-core:Triaged> <https://launchpad.net/bugs/1534627>15:47
mupBug #1534626 opened: provisionerSuite.SetUpTest fails because connection reset <ci> <intermittent-failure> <reliability> <juju-core:Triaged> <https://launchpad.net/bugs/1534626>15:50
mupBug #1534627 opened: Destroyed models still show up in list-models <juju-core:Triaged> <https://launchpad.net/bugs/1534627>15:50
mupBug # opened: 1534632, 1534636, 1534637, 153464315:53
mupBug # changed: 1534632, 1534636, 1534637, 153464315:56
mupBug # opened: 1534632, 1534636, 1534637, 153464315:59
cheryljcan I get another easy review?  http://reviews.vapour.ws/r/3549/16:21
natefinchcherylj: ship it16:30
cheryljthanks, natefinch!16:30
natefinchkatco, ericsnow: someone needs to make a "let me man pages that for you"17:05
natefinchfwereade: I don't suppose you have a test charm that exercises leadership?17:07
lazyPowernatefinch: i have one17:50
mupBug #1534757 opened: Attempting to run charm before unit provisioned, 1.26 <juju-core:New> <https://launchpad.net/bugs/1534757>17:57
natefinchlazyPower: I actually figured out that I don't need a special charm... every charm gets a leader, most charms just don't care :)18:08
natefinchlazyPower: so I can just use a dummy charm and juju run is-leader on it, and that's sufficient for my purposes :)18:08
beisnermgz, cherylj - still re-confirming, but two bootstrap attempts with 1.25.2 + 1.9 give me a bootstrap timeout.  the machine comes online, maas state is deployed.  it's reachable for a bit, then no-talky from then on.  http://pastebin.ubuntu.com/14506872/18:21
mgzbeisner: thanks18:31
mgzbeisner: what does maas say about it?18:32
beisnermgz from maas perspective, it is "deployed"18:32
beisneri've backed out to 1.25.0, same procedure, same maas, bootstraps ok18:33
beisneri suspect juju bridge foo is the moment we lose network18:33
beisnerunfortunately, i can't inspect the aftermath b/c i can't reach it.  i've got SOL / ipmi, and can see that it is up at the login screen.  but of course no user/pwd login enabled.18:34
mgzcan you ssh in parallel when it's installing and stuff and hold that connection?18:34
mgzI'd be interested to see logs/networking config changes on the box18:35
beisnerme too.  i shall keep trying.    do you have 1.9 maas that you can try a bootstrap against?18:35
mgzbeiser: see http://reports.vapour.ws/releases/3511/job/maas-1_9-OS-deployer/attempt/18118:37
mgzlog in with sso and go to that page18:37
mgzthat's our result for 1.25.2 on maas 1.9 deploying your trusty-liberty bundle18:38
beisnermgz, that's using 1.25.318:40
beisneri'm using 1.25.2 from ppa:juju/proposed, plus agent-stream: proposed18:40
mgzah, whoops, wrong rev, go back to /releases/3506 and attempt/16318:41
mgzwhich is the same code, though not literally the same binaries18:41
mgzour maas is certainly not your maas though18:42
mgzbeisner: I'm presuming you don't have any of the feature flags on as well?18:42
beisnermgz right, just as a vanilla new user18:42
beisnerok, i'll fire off a fresh bootstrap with 1.25.2 and see if i can sneak onto that unit before it goes lalalala18:43
mgzas soon as it starts the apt installing stuff you should be good, note you'll want the same ssh key/juju env key available18:44
beisnerha.  i got this seconds before the first reboot.  http://pastebin.ubuntu.com/14507351/    /me watches closely...18:56
beisnerok mgz.  i had a total of ~10s of reachability, then unreachable.  i did establish ssh in that window, did another `ip a` and got as far as the b in `brctl show` when it cut me off:19:01
beisnerhttp://pastebin.ubuntu.com/14507430/19:01
beisnerso before maas' first reboot, sane.19:01
beisnerafter maas' first reboot, sane.19:01
beisnerafter juju init stuff, no connectivity.19:02
perrito666oh great someone did the same I was doing19:02
mgzso, session did get cut off, that pretty much does mean this is a interface reconfiguration19:03
katconatefinch: planning19:05
natefinchkatco: oops, coming19:06
mgzbeisner: I think we need a bug to track. probably want maas network output included.19:07
beisnermgz, yep i'm raising one now.19:08
frobwarebeisner, can I give you a binary that would allow console login19:11
beisnerfrobware, yes please & thx19:11
frobwarebeisner, you would obviously have to bootstrap with --upload-tools19:12
beisnerfyi:  bug 153479519:12
mupBug #1534795: unit loses network connectivity during bootstrap: juju 1.25.2 + maas 1.9 <maas-provider> <uosci> <juju-core:New> <https://launchpad.net/bugs/1534795>19:12
beisnerfrobware, ack19:12
frobwarebeisner, ok give me 10-15mins - will verify locally19:13
beisnerthx frobware19:13
beisnermgz, frobware - good news:  in our openstack-provider automation, things are looking good so far :)19:15
frobwarebeisner, so any clue as to why the other setup is borked? different MAAS setup?19:15
beisnerfrobware, two different providers.19:15
mgzfrobware: using the maas provider rather than openstack provider19:15
frobwarebeisner, on the MAAS side what's the network config look like? VLANs? Bonds? straight-forward eth0?19:16
frobwarebeisner, and version of MAAS?19:17
beisnermaas:  1.9.0+bzr4533-0ubuntu1~trusty119:18
beisnerjuju:  1.25.2-0ubuntu1~14.04.1~juju119:18
frobwarebeisner, ack19:18
beisnerfrobware, eth0, untagged19:19
frobwarebeisner, so as vanilla as it gets.... sigh.19:19
frobwarebeisner, amd64?19:19
beisneryep amd6419:19
frobwarebeisner, just checking. ;-)19:19
beisnerfrobware, yep, np19:20
frobwarebeisner, what's the easiest way to get binaries to you?19:21
beisneri've used our private chinstrap in the past for that.  i've not used the new private-fileshare bits yet, though we can try that.19:22
beisnerfrobware ^19:22
frobwarebeisner, any other interfaces or just eth0?19:26
mupBug #1534795 opened: unit loses network connectivity during bootstrap: juju 1.25.2 + maas 1.9 <maas-provider> <uosci> <juju-core:New> <https://launchpad.net/bugs/1534795>19:27
beisnerfrobware, machine has 4 nics, only eth0 and eth1 are cabled up.  we've only ever used eth0 on these particular machines.19:27
beisnerfrobware, mgz:  oo interesting ... note that on my 2nd paste in the bug, eth1 is up for some reason.19:29
beisnerthat is unexpected19:29
frobwarebeisner, http://178.62.20.154/~aim/juju-bin.tar.bz219:35
frobwarebeisner, that's 1.25.2 but before juju has a little play with /e/n/i is does `passwd -d ubuntu' - so you should be able to login on the console.19:36
frobwaremgz, ^^19:37
mgzfrobware: thanks for helping out19:37
beisnerthx, grabbing the bin now frobware19:38
frobwaremgz, beisner: I'm very tempted to make that a capability behind a flag... as we run into it too often and it's difficult to diagnose once it goes wrong.19:43
frobwaremgz, beisner: the issue of course is security19:43
beisnerthat would be a good t-shooting tool indeed.   but yep, that.19:43
frobwarebeisner, I just tried with two NICs configured and it bridged eth0 OK19:47
beisnerfrobware, so i have two nics CONNECTED, but not two nics configured.19:48
beisnerfrobware, ie.  http://pastebin.ubuntu.com/14508207/19:48
frobwarebeisner, so when I say configure, in MAAS 1.9 parlance, they're configured and both have static ip assignment when looking in the MAAS UI19:48
beisneryet eth1 winds up UP19:48
beisnerfrobware, https://launchpadlibrarian.net/234276622/network.png19:52
frobwarebeisner, let's see what happened to e/n/i once you bootstrap and can login19:52
frobwarebeisner, but I see no smoking gun there19:53
beisnerfrobware, \o/  i'm in via ipmi19:53
beisnersol that is19:53
beisnerwith that bin19:53
frobwarebeisner, the binary I sent you didn't work?19:55
beisnerfrobware, yes.  it let me log in with ubuntu.  now poking around in it.19:55
beisnerfrobware, eth0 is down;  eth1 is up;  no bridges.  no ipv4 addresses.19:59
beisnergrabbing /etc, /var/log, anything else?19:59
frobwarebeisner, can you send me /etc/network/interfaces*19:59
mupBug #1534804 opened: debug-log is unstable with LXD local provider <docteam> <juju-core:New> <https://launchpad.net/bugs/1534804>20:00
frobwarebeisner, and capture ip route20:00
frobwarebeisner, is is possible to HO?20:01
beisnerfrobware, https://plus.google.com/hangouts/_/canonical.com/andrew-mcdermot?authuser=120:06
beisnerwow `ip route` blank20:07
beisnerthat's a new one20:07
frobwarebeisner, in there20:07
frobwarebeisner, mgz: http://pastebin.ubuntu.com/14509474/20:19
beisnerfrobware, added tarball to bug. woot.  thx again for your help.  & mgz20:35
frobwarebeisner, mgz: I know exactly what the bug is.20:39
frobwarebeisner, mgz: fixed in maas-spaces, doesn't help... however, we ran into the same issue a week ago on maas-spaces - and we had not seen it there for > 1 month. Something changed upstream so that we get dns-nameserver entries partwya through /e/n/i and we don't parse that correctly.20:40
perrito666:| ubuntu does not define xdg_config_home ?20:41
beisnerfrobware, excellent.  and yep, i'll do a maas non-juju deploy of that machine and collect the same in a bit.20:42
frobwarebeisner, pretty sure you won't need to do that. uploading a new binary for you to try.20:43
frobwarebeisner, I'll leave the `passwd -d ubuntu' in just to help out.20:45
frobwarebeisner, I still don't understand why my MAAS setup doesn't give me the same dns-nameserver entries other people get.20:46
cheryljfrobware: I'm really happy you're still up to look at this :)  I saw the bug report while at the doctor's office20:49
frobwarebeisner, http://178.62.20.154/~aim/juju-bin.tar.bz220:52
frobwarebeisner, md5sum a2d16e6f3993cc56e7c998aadfe48b7420:52
beisnerfrobware, thx.  will grab and run it shortly20:54
frobwarecherylj, https://bugs.launchpad.net/juju-core/+bug/1534795/comments/721:36
mupBug #1534795: unit loses network connectivity during bootstrap: juju 1.25.2 + maas 1.9 <maas-provider> <uosci> <juju-core:Confirmed for frobware> <https://launchpad.net/bugs/1534795>21:36
frobwarecherylj, https://bugs.launchpad.net/juju-core/+bug/1534795/comments/821:36
frobwarebeisner, see my updates to the bug ^^21:37
cheryljfrobware: thanks so much!21:37
beisnerfrobware, cherylj - cycling now.  it's progressed to pkg installation and maintained connectivity thus far \o/21:38
frobwarebeisner, great. I would really appreciate if you could try it again with http://178.62.20.154/~aim/juju-bin-lp1534795.tar.bz2 as this is a build from the proposed PR.21:38
frobwarebeisner, I'll hang around for 10-15 mins to hear your "BOOTSTRAPPED!" message. :)21:39
frobwarecherylj, beisner: the only thing I would point out about my binary build is it is based on the tip of 1.25... not the 1.25.2 tag. Caveat Emptor!21:42
katcoericsnow: natefinch: what was your alternative proposal for "juju charm --resources"?21:45
natefinchkatco, ericsnow: anything else21:48
natefinchkatco, ericsnow: uh... maybe just charm-resources?21:48
ericsnownatefinch: I actually think it fits well, even if it requires a new juju sub-command21:49
natefinchericsnow: I can't abide adding a new subcommand that doesn't do anything, just so it can have a flag that does something.  That's terrible.21:49
ericsnowkatco, natefinch: yeah, ideally charm-resources21:49
katcoericsnow: natefinch: ok, i'll float the suggestion. it's entirely possible that we intend to expand on the charm command21:52
natefinchkatco: I'd be fine with it if juju charm was a command that was goign to do other things.... though I'm generally against using flags to do completely different things... that's what subcommands are for21:53
natefinchkatco: so like, if juju charm cs:foo gave you generic info about the charm, and juju charm --resources gave you info about the charm's resources, that's cool21:54
perrito666please dont add an empty command, that is counter intuitive21:54
katcoperrito666: we're fighting it21:54
perrito666charm-resources sounds a bit less painful21:55
perrito666my kingdom for an unblocked master21:56
natefinchditto21:56
beisnerfrobware, 0          started 1.25.2.1 fat-machine.dellstack21:57
katcoperrito666: natefinch: it's not going to unblock until cherylj decides we're good for stabilizing 2.0-alpha1: http://pad.lv/153375021:57
natefinchlooks like just this left: https://bugs.launchpad.net/juju-core/+bug/153420121:58
mupBug #1534201: TestHelpAddImage fails <ci> <regression> <test-failure> <windows> <juju-core:In Progress by cherylj> <https://launchpad.net/bugs/1534201>21:58
cheryljyeah, I need to add back in the feature flag without breaking windows21:59
anastasiamaccherylj: it's staurday here but.. i can look at it at my monday21:59
anastasiamaccherylj: did u get a chance to progress this bug?21:59
frobwarebeisner, that's booted OK then?22:00
natefinchlater all, dinner time22:00
cheryljanastasiamac: I haven't yet.  going to this evening22:00
cheryljanastasiamac: did you have anything to hand off?22:00
mupBug #1532777 changed: undertakerSuite.TestEnvironConfig fails <blocker> <ci> <intermittent-failure> <regression> <test-failure> <juju-core:Fix Released by waigani> <https://launchpad.net/bugs/1532777>22:00
anastasiamaccherylj: k.. keep me posted22:00
anastasiamaccherylj:  jsut an idea on how to fix it from tim...22:00
anastasiamaccherylj: the fundamental problem is tests should not touch external singleton resources, such as the windows registry22:01
cheryljyeah...22:01
anastasiamaccherylj: this is why all the tests use the environment for feature flags22:01
anastasiamaccherylj: i suspect that the base test suite for these commands needs to be re-written22:02
anastasiamaccherylj: but i was going to see if i can re-run juju tests on my wndows22:03
anastasiamaccherylj: and had trouble getting it going22:03
mupBug #1532777 opened: undertakerSuite.TestEnvironConfig fails <blocker> <ci> <intermittent-failure> <regression> <test-failure> <juju-core:Fix Released by waigani> <https://launchpad.net/bugs/1532777>22:03
cheryljanastasiamac: yeah, thanks to sinzui, I have a windows machine I can use for unit tests22:03
anastasiamaccherylj: \o/22:04
perrito666oh, blessed master22:04
anastasiamaccherylj: neeed to go to have a weekend :/ will check back in on monday and pick it up if needed22:05
sinzuicherylj: master is reightfully bless. I parused CI because I am looking to move some jobs to other public clouds since Joyent hasn't fixed the firewall yet22:05
beisnerfrobware, yessir22:05
cheryljthanks anastasiamac!  have a good weekend!22:05
anastasiamaccherylj: thanks \o/ you as well!22:05
frobwarebeisner, excellent.22:06
* frobware is gone gone gone!22:06
cheryljthanks, frobware!22:07
mupBug #1532777 changed: undertakerSuite.TestEnvironConfig fails <blocker> <ci> <intermittent-failure> <regression> <test-failure> <juju-core:Fix Released by waigani> <https://launchpad.net/bugs/1532777>22:09
perrito666oh great, juju home is not defined by a single source of truth22:09
beisnercherylj, frobware, mgz - ok i'm eod + eow.  off monday, US holiday.   i can commit to cycling the 3rd bin (http://178.62.20.154/~aim/juju-bin-lp1534795.tar.bz2) at some point this weekend and chiming in on the bug with how that went.22:20
cheryljthanks, beisner!22:21
perrito666ok, this got into my nerves, EOW22:23
beisneryw, cherylj    o/  have a nice weekend, all22:23
=== ericsnow is now known as ericsnow-afk

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!