[00:06] smoser, how odd. is there anything I can do to debug this further? [00:07] smoser, I don't think it's using the daily image, MAAS images have been configured with maas-import-pxe-files [00:11] matsubara, can you get console logs ? [00:12] i'd like to see them. [00:12] what version of maas are you using ? [00:12] smoser, I'm looking at the console through the Raritan java client but it doesn't seem to allow me to change the tty or boot into recovery mode [00:13] smoser, I'm using the latest package from the daily builds archive: maas - 0.1+bzr1002+dfsg-0+1007+87~ppa0~quantal1 [00:21] matsubara, can you get ttyS0 logs ? [00:21] (serial console) [00:21] smoser, how could I get those? [00:21] what made you think yo uwere waiting on eth1 ? [00:21] well, its possible there is serial over ethernet or some way to get at a serial console. [00:22] the console is halted at the eth0: link become ready [00:22] smoser, I can show you an screenshot, just a sec [00:29] matsubara, ok. if the console is all you have to debug, and you can't get a serial console, then you'll want to edit one of the maas files and remove 'console=ttyS0' [00:30] smoser, https://dl.dropbox.com/u/551281/node-eth-ready.png [00:31] easiest thing to do is just add 'console=tty1' to all of the APPEND lines in the template files in /usr/lib/python2.7/dist-packages/provisioningserver/pxe/ [00:32] that will make your graphical console the place where messages get written. [00:32] i made the change to use serial console [00:32] which, honestly, i think is right. [00:32] no one will ever have text logs of a graphical console [00:33] matsubara, i dont think anything there is unexpected. [00:33] you're just not seeing output becausae its going to the serial console. [00:33] did you follow the above ? [00:33] how to change the console= ? [00:33] matsubara, ^ [00:33] smoser, yes, let me change those files [00:34] matsubara, you gonna be around a while? [00:34] i have got to step away for 20 minutes at least. [00:34] smoser, like this: APPEND {{kernel_params(arch="amd64") | kernel_command | console=tty1}} ? [00:35] but i can come back or we can pick up tomorrow. [00:35] i'd just add it outside the }} [00:35] smoser, yes, I'll be around [00:35] APPEND {{kernel_params(arch="amd64") | kernel_command}} console=tty1 [00:35] (basically just append that string to the end of those 'APPEND' lines) [00:36] smoser, cool. did that and am rebooting the node [00:39] smoser, https://dl.dropbox.com/u/551281/node-eth-ready%3Dconsol-tty1.png [00:40] so I guess what's failing is maas-enlist and not the network interface readyness [01:06] matsubara, you should get mor output than that. [01:06] hm.. [01:06] smoser, that's all I got with that change [01:06] i'm not sure what would be saying 'hostname' like that. [01:06] hm.. [01:07] the hostname is 'maas-enlist' because that is what the kernel cmdline says to set it to. [01:07] smoser, I did notice that console=tty0 was still in the parameters for the kernel when the node booted [01:07] hm.. [01:07] "no response after 2 seconds" giving up [01:07] i didnt think i saw that bfore [01:07] oh, wait, but it does retry, i'm almost certain of that. [01:07] hm.. [01:08] roaksoax, is there anyway we can get ttyS0 output in that lab? [01:08] ipmi or something? [01:11] matsubara, is there any doc on that lab? i've not played in it. [01:11] smoser, https://wiki.canonical.com/Launchpad/QA/MAASLenovoLab [01:13] matsubara, you added 'console=tty1' to each of those files ? [01:13] you should see hte kernel cmdline printed to the console [01:13] and you can verify that the last thing on it should be 'console=tty1' [01:13] we can also add '--verbose' and 'debug=1 [01:14] and that will give you no shortage of data [01:14] smoser, I added to both APPEND lines. let me add those two other options [01:14] which file ? [01:14] /usr/lib/python2.7/dist-packages/provisioningserver/pxe/ [01:14] /usr/lib/python2.7/dist-packages/provisioningserver/pxe/config.commissioning.template [01:14] sorry, yes [01:14] is probably the rigth one for enlistment and commissioning [01:15] rebooting [01:20] smoser, https://dl.dropbox.com/u/551281/node.png [01:41] matsubara, can you just let that sit there? it should get furhter eventually. [01:41] something else shoudl happen. it might be 300+ seconds. but *something* should happen [01:55] matsubara, well... good news and bad news. [01:55] good news, i just successfully enlistsed a node following http://bazaar.launchpad.net/~smoser/maas/maas-pkg-test/view/head:/maas-ephemeral-test-quantal.txt [01:56] without using the 'daily' ephemeral image (which is what you were doing). [01:56] matsubara, ah. i wonder. [01:56] did you knwo that you have to fix dhcp ? [01:56] h.. [01:56] hm.. [01:56] i'll look more at this tomrrow with you if thats ok. [01:57] but i'm not really sure what i could have regressed. [02:20] smoser, what fix to dhcp? [02:21] smoser, if I let it sit there it stays there, doesn't seem to time out [02:29] smoser: yeah ipmi should be your best bet if those have ipmi [02:33] her [02:39] well, in my setup next-server got set incorrectly. [02:39] matsubara, [02:39] so i had to fix 'next-server' in /etc/dhcp/dhcpd.conf [02:40] smoser, right, I fixed that running dpkg-reconfigure maas-dhcp [02:40] that wouldn't get fixed by that. [02:40] i dont think [02:40] or if it does, then i misunderstand the problem. [02:41] smoser, I ran dpkg-reconfigure maas and then dpkg-reconfigure maas-dhcp to get the proper dhcpd.conf written [02:42] ah. ok. so then you have dhcpd running on the same interface as maas-server [02:42] (ie, eth0 for you probably) [02:42] (meanwhile, any new information on the “cp” problems in maas-import-ephemerals?) [02:42] smoser, yes [02:44] matsubara, how long did you let it sit at that ? and did you do so with --verbose and debug=1? [02:44] you may have to let it set for 5 minutes. [02:44] but something should time out. [02:44] smoser, nope, I let it sit for awhile, I guess more than 5 min. I can try it again [02:48] matsubara, the other thing we can do is take the 'console=ttyS0' out of /usr/share/pyshared/provisioningserver/kernel_opts.py and restart maas-pserv [02:48] but i really dont think thats going to have an affect. [02:49] the only other guess i have is that I changed 'ip=dhcp' to 'ip=::::' [02:49] but my test here verified that that generally works even on the precise image. [02:50] to be clear, I changed kernel paramters, adding 'console=ttyS0' [02:50] smoser: sorry to interrupt — maybe you thought my questions were rhetorical or something, but I really need to know. Is it only for our own dhcpd instance that the leases file needs to be root-owned? If so, how can we work around that? If not, how does the regular dhcpd write to it? [02:50] and 'console=tty1' [02:50] its only the leases file. [02:50] Stop. [02:50] it opens it for wrigint before dropping permissions is what I suspect [02:50] I am asking a question about the leases file. [02:50] You answer with “it's only the leases file.” [02:50] I think we're still talking at cross purposes. [02:51] "regular dhcpd" writes that file as root (look at /etc/init/dhcp-isc-server.conf) [02:51] ! [02:51] we need to do exactly the same thing. [02:51] But the dhcpd doesn't run as root, does it? [02:52] it opens it for wrigint before dropping permissions is what I suspect [02:52] What is wrigint? [02:52] writing [02:52] Then don't type it wrong twice! [02:53] If you repeat an obvious mistake, you reinforce the notion that you're doing it deliberately — leaving me with the puzzle of what you mean if not the obvious. [02:53] sorry. [02:53] but if you looked at /etc/init/dhcp-isc-server.conf or actually played with this, you'd see. [02:54] I did look at dhcpd-isc-server.conf… and in fact it's where I copied my own version from. [02:54] http://paste.ubuntu.com/1212215/ [02:55] chown root:root /var/lib/dhcp /var/lib/dhcp/dhcpd.leases [02:55] chown root:root /var/lib/dhcp/dhcpd.leases~ [02:55] where yours did: [02:55] chown maas:dhcp $LFILE [02:55] chmod g+w $LFILE [02:56] Interesting — I wonder if I just missed that comment. [02:56] Nope: the comment is not there in the version of isc-dhcp-server that I have installed! [02:56] smoser, not sure if it changes anything but I'm running this on quantal (I upgraded the QA lab to quantal last Friday) [02:57] The version I get installed chowns the leases file to dhcpd:dhcpd. [02:57] matsubara, well, it should not hcange anything. [02:57] So this looks like a Quantal change. [02:58] What I did was apparently right for Precise, and what you ran into was new to Quantal. I wonder why they made that change. [03:05] it would seem potentially a change in behavior from upstream 4.1.ESV-R4 -> 4.2.2 [03:13] jtv: cause it is usually better to run a daemon as a user and not as root [03:16] smoser, apparently it doesn't time out. it's been stuck there for more than 5 min [03:17] smoser, last message is eth1: no IPv6 routers present [03:17] thats random. [03:17] it seems to me like you're not getting a dhcp response. [03:17] or you got one on the wrong interface. [03:17] but i really think the former. [03:18] roaksoax, it runs as non-root in precise also. [03:18] it really seems like a bug (regression) to me that their code now opens the leases file before it drops its permissions. [03:18] but, that is the way it is, and we'll just have to deal with that or come up with some solution that works both on precise and quantal. [03:19] matsubara, see above. again, i can poke at this some tomorrow. [03:19] smoser, ok. I'll ping you. thanks! [03:20] it's strange because it does get the IP adress and gets the image from the tftp server [03:20] smoser: most likely, the changelog doesn't seem to provide any hint of the change on permission for the leases file though [03:20] upstream changelog? [03:21] the ubuntu bzr log is screwed up due to the importer sucking [03:21] i already blamed mdlesuer for something that wasn't his fault as a result :) [03:22] heh [03:22] smoser: i'm checking on debian's changelog [03:23] roaksoax: that's the funny thing — dhcpd was _already_ running as its own user. The change I'm wondering about is that the leases file must now apparently be owned by root rather than by the one user that's supposed to read and write the file. [03:24] jtv: right, so I'm wondering if this was due to a recent upload, and if so, have you identified which one is it?": [03:26] jtv: this could also be related to the fact that we are running our own dhcp server too [03:27] roaksoax: those are exactly the answers I've been trying to get out of Scott. And now it finally turns out that he's been looking at a different version of the package than I got. [03:28] lol [03:28] It turns out that the leases file needs to be root-owned _in the regular Quantal isc-dhcp-server package_. [03:29] Not funny — cost me days off this end of my life and probably a few years off the other end. “Why don't you just look at the file? It's so simple. I'm not going to answer direct questions.” [03:29] Well in retrospect maybe it is a little funny. [03:30] Anyway, the Quantal package's upstart script chowns the leases file to root, and has a comment saying that this is required. The precise version didn't have any of that. [03:31] So it's not related to the fact that we're running our own instance — it just complicates running our own instance. [03:32] jtv: right, but if we are running our own version, it still meeans we assign permissions as maas:root or maas:dhcp [03:33] Well that's the thing: _something_ in the package seems to be requiring root:root, so apparently it has to be root:root. [03:34] In Quantal. [03:34] But making it root-owned may actually break the Precise version. [03:36] jtv: right, so it is simply different permissions for each verison of the package [03:36] Can we do that easily? [03:39] roaksoax: does that mean I should just propose separate (but similar) branches for the packaging and packaging-precise branches? [03:40] jtv: the precise packaging branch has changes stacked on top of the quantal branch [03:40] jtv: so the process should be propose to quantal [03:40] then apply precise changes and commit to precise [03:40] or propose to precise too [03:41] OK I'll focus on getting it working on quantal then. [03:43] jtv: ok, let me know if I can help with anything (of course i'd take care of it tomorrow_ [03:43] jtv: or just drop me an email with what's needed to get changed and I will take a look at it tomrorow [03:43] now i'm off to bed [03:43] Thanks and good night! === matsubara is now known as matsubara-afk === matsubara-afk is now known as matsubara [14:12] allenap: I have an issue with gen_pxe_template_filenames. I need an ARM-specific version of config.template, but this isn't currently possible. See http://bazaar.launchpad.net/~maas-maintainers/maas/trunk/view/head:/src/provisioningserver/pxe/config.py#L45 [14:12] allenap: can we get rid of config.template altogether and require every template to have a purpose? [14:12] alexmoldovan: or alternatively have a config.template.{arch} fallback before config.template? [14:13] uh, allenap [14:15] rbasak: First thing, the docstring lies; config.{purpose}.template will always be tried before config.template. [14:15] rbasak: Ignore that. [14:15] I am very tired right now :) [14:16] rbasak: If you mean you want the same template for multiple purposes, then I suggest using symlinks. [14:16] rbasak: These already exist for other templates. [14:17] Where these equals symlinks. [14:17] I don't think that solves my problem [14:17] I need config.template to be different on ARM [14:18] rbasak: config.arm.template with symlinks for each purpose? [14:18] So we either need to drop config.template and use specific purpose templates only (in which case I can use the existing override), or we need to add an extra override [14:18] I don't really understand the problem. [14:18] We could do that, but then it wouldn't be parallel to x86 [14:18] If we do that, then we want config.template with x86 symlinks for each purpose too [14:19] I want to do the same as what's there already with just the exception for arm present, rather than arranging everythign there differently for arm and thus confusing future work [14:23] rbasak: I don't think that's a problem - not being symmetrical - but I think I still don't really see what the issue is. Another way to do it might be: [14:24] get rid of the {purpose} lookup, and instead pass purpose in to the template's namespace. Then the template can choose what to render. [14:27] rbasak: I think that actually makes more sense in terms of making these templates useful for multiple architectures. [14:27] Enabling a new piece of hardware would mean providing templates for all purposes on its arch, not just a subset. [14:28] allenap: that's a fairly hefty change and I'm not sure what benefit it would give us given where we are right now. We don't know what'll happen to the boot purposes in the end either, right? [14:28] allenap: I think I'd prefer to add a fallback to config..template which I know won't break anything [14:29] (and is a trivial change) [14:31] rbasak: Yeah, I still don't see why it's needed. It complicates the lookup, which can already be hard to follow. Just use symlinks for now. There are three boot purposes to make links for : commissioning, install, and local. [14:33] allenap: so we already have a config.commissioning.template and a config.local.template [14:33] allenap: can I just rename the existing config.template to config.install.template then? [14:34] allenap: then I can have easy armhf overrides as necessary [14:35] rbasak: Yeah, that makes sense. That was your original suggestion too. Sorry, I'm not in a fit state to field questions today! [14:36] allenap: no worries. I wasn't sure if there were other boot purposes too before. I'll test and get a merge proposal in. [14:37] rbasak: get_boot_purpose can also return "poweroff", but this isn't handled anywhere. That's returned for nodes that really ought not to be booting. [14:38] allenap: is that just an fyi or will it affect my proposed change? [14:38] AIUI, I can ignore that? [14:38] rbasak: fyi [14:38] Yeah. [14:38] can I mark my own mps approved if they have an approve vote and I've done the requested changes? [14:38] Right. Thanks! [14:39] mgz: Yes. [14:39] allenap: ta [14:39] mgz: For small changes you can +1 and land your own mps too if waiting for a review is going to block you. [14:39] mgz: don't forget to set the commit message otherwise the MP won't be picked up by tarmac. [14:40] rvba: also ta [14:40] "[bug=]"... seriously? [14:43] not pulling the author/bug details from bzr is bad enough, but we want to pander to commit log parsers so dumb they can't cope with a missing field? [14:44] or is tarmac adding that? [14:44] Yep, tarmac does that. [14:45] okay, /me bops tarmac on the head for being silly [15:51] this make any sense to anyone [15:51] https://bugs.launchpad.net/maas/+bug/1051626 [15:51] Ubuntu bug 1051626 in MAAS "next-server written wrong" [Critical,Confirmed] [15:51] see my last comment there. i'm not sure what is making the pxe boot take a different path [15:52] allenap, where is your maas cli branch ? [15:52] never mind [15:52] https://code.launchpad.net/~allenap/maas/command-line/+merge/124704 [16:00] rvba: btw.. were you able to look on the release support branch? [16:03] smoser: Weird indeed! [16:04] roaksoax: we talked about it this morning during the standup. Jeroen pointed out that you should use the new model BootImage, instead of getting the list of releases from the command line tool. [16:04] rvba: we are not getting the releases from a command line tool [16:04] rvba: the realeases are being get from python-distro-info === matsubara is now known as matsubara-lunch [16:05] roaksoax: ah right, I see that now. I was just wonder were that UbuntuDistroInfo was coming from. [16:05] roaksoax: still, BootImage should probably be used instead. [16:05] rvba: what's that now ? [16:06] roaksoax: src/maasserver/models/bootimage.py [16:07] rvba: right, so hold on, how we want to obtain the releases from what boot images we have available, since we should obtain the bootimages from the releaes that we have available [16:07] rvba: i mean... we should obtain images from release, not release from images available [16:08] allenap, at the risk of seeming completely stupid. [16:08] how can i play wiht maascli [16:08] jtv: still around? [16:09] roaksoax: maas-import-pxe-files should populate BootImage. [16:09] and maas-import-pxe-files uses distro-info. [16:09] smoser: Branch lp:~allenap/maas/command-line, make, then use bin/maascli api login --help to get started. [16:10] smoser: It hasn't landed yet :-/ [16:10] rvba: right, but IMHO, maas-import-pxe-files should obtain the images from the releasea MAAS knows about, not the other way around [16:10] allenap, well, 'make bin/maascli' was failing for me [16:10] i had'nt 'amke install-dependencies' yet [16:10] rvba: so maas-import-pxe-files should obtain the releases tht MAAS support [16:10] rvba: so maas should determine what releases it supports, not a helper script [16:10] but truthfully, the point of 'make' is that it would have recognized i'd not done that :) [16:11] plus, i already have maas installed on the system via package. i hate pypy [16:11] roaksoax: your suggestion makes sense to me. [16:11] anyway, i'm getting there. [16:11] rvba: ok, so we really need to get something in asap as it is really delaying my work [16:12] rvba: improvements can always con after we have the main parts in [16:12] such as better determining the avaalble releases [16:12] roaksoax: granted. So my suggestion is this: hardcode the available versions for now: XXX = ['precise', 'quantal'] [16:13] roaksoax: the rest can stay as is. [16:16] rvba: OK [16:16] rvba: do you also want split braches? [16:17] roaksoax: you could simply split that branch into two: one for the db change and one for the rest but I'm ok with one branch if that's easier for you. [16:17] rbasak, lp:~smoser/maas/maas.ubuntu.com.images-ephemeral now has my pushed branch [16:17] rvba: i'll split it for better understanding [16:18] i hadn't updated previously. i thought i had pushed. [16:18] :-( [16:18] rvba: though, will it get merged before having all the tests addressed? [16:18] smoser: I'm having trouble building your PPA mountall with sbuild [16:18] roaksoax: usually, the tests should be done in the same branch. [16:19] rbasak, why? [16:19] (fwiw, what is there is == to what is in lp:ubuntu/mountall at the moment) [16:19] rvba: ok, so that means fixing looooooooooots of them [16:19] smoser: http://paste.ubuntu.com/1213219/ [16:19] but note, that lp:ubuntu/mountal != whats in the archive. [16:20] smoser: looks like it wants plymouth-dev instead of libplymouth-dev [16:20] smoser: http://osdir.com/ml/general/2012-03/msg44302.html looks like the same problem [16:20] roaksoax: really? Since there is a default release, I don't expect this to break that many tests… [16:22] rvba: the new attribute brekas them I think [16:22] rvba: i'll have to recheck [16:24] rbasak, you're trying to build in quantal, right? [16:24] yeah, looks like it. [16:24] strange. [16:24] in a quantal chroot, in precise [16:24] I found --resolve-alternatives [16:24] trying it now [16:25] roaksoax: only 2 failures. We should manage :) [16:25] OK that seems to have worked [16:25] ok cool [16:25] lol [16:25] Looks like it's intended sbuild behaviour === cmagina is now known as cmagina_away [16:26] rvba: alright, I'll do the changes and re-submit the merge proposal(s) [16:26] roaksoax: cool. One more thing, let's set the default to be '' instead of None so won't have to check for both every time we want to know if the value is empty. [16:26] rvba: ok cool [16:28] smoser: http://people.canonical.com/~rbasak/mountall_2.41~ppa415_armhf.deb but I can loopback mount the ephemeral image once it's installed to stick that in, right? And you want me to regenerate an image from lp:~smoser/maas/maas.ubuntu.com.images-ephemeral and test with that? [16:30] rbasak, yeah, pull the m.u.c branch. then edit ARCHES to put armhf in it. [16:30] and run [16:31] or, you dont really need all the launcher stuff [16:31] just download the .tar.gz file, extract and then run [16:31] ./maas-cloudimg2ephemeral disk.img arm-kernel arm-initrd arm-manifest [16:31] and it should handle it [16:32] (then mount loopback and dpkg -i the mountall) [16:32] What .tar.gz file? Will a branch checkout do? [16:33] smoser: ^^ [16:37] working on it [16:46] rbasak, http://paste.ubuntu.com/1213255/ [16:46] it looks a lot like that [16:46] but then add the moutnall [16:47] and that is not working for me on amd64 on quantal right now [16:47] as the apt-add-repository is failing [16:48] smoser: I'm not with you [16:48] no? [16:48] smoser: we're trying to get me to generate an armhf ephemeral image, right? [16:48] yes [16:48] what's nelson? [16:48] my local '$mirror' [16:48] what can I use instead? [16:49] cloud-images.ubuntu.com [16:49] cloud-images.u.c? [16:49] ok [16:49] smoser: so I don't run the command on the first line, right? === cmagina_away is now known as cmagina [17:01] rbasak, shoot. [17:01] right. [17:01] do not run the first line [17:01] only subsequent [17:01] bad paste [17:01] ok [17:01] almost there [17:03] discovering some dependencies - qemu-user-static and rsync [17:03] http://paste.ubuntu.com/1213282/ [17:03] that looks a bit better [17:03] you should not need qemu-user-static [17:03] if you're doing this on arm [17:03] I'm not [17:03] and rsync? [17:03] A script seemed to use rsync [17:03] i dont think so [17:03] I'm doing this on my test maas machine, which is virtualised amd64 [17:03] ah. you're right . tree2query does [17:06] rbasak, but for this exercise you dont need tree2query [17:06] smoser: it failed without rsync [17:07] you're nto running tree2query [17:07] * rbasak shrugs [17:08] shoot [17:08] and if you're hitting that cod then something else is wrong [17:10] rbasak, shoot. thats a bug. let me fix that. [17:10] i'm sorry [17:11] it's ok [17:11] I can cope with needing to install rsync :) [17:12] smoser: dd: writing `/tmp/maas-cloudimg2ephemeral.LKQPwv/mp.K8gUX1/tmp/zero': No space left on device [17:12] is that a problem? [17:12] or intentional to zero out unused space? [17:12] intentional [17:13] OK so I think I have an image [17:13] in ~/dl? [17:14] smoser: now what? [17:14] smoser: I don't understand the different forms these images can take [17:15] This doesn't look like something that maas-import-ephemerals can read, and I don't see an initrd.gz [17:16] rbasak, it updates $img in place [17:16] and copied out kernel and intiramfs to the places you told it to [17:16] Ah [17:16] dont use maas-import-ephemerals, just copy those files to the right places. [17:16] OK I think I follow === matsubara-lunch is now known as matsubara [17:25] rbasak, 39. i think is reasonable. [17:25] r39 in that branch. [17:25] almost finished here. [17:26] rvba, why do you want to hardcode a list of releases? [17:27] we've solved this problem in distro with 'distro-info' and it magically gets serviced for you. [17:27] smoser: it's just a temporary measure. We want a central place to store that information. [17:28] temporary as in how long? [17:28] if temporary as in its-going-to-go-in-12.10 [17:28] then use distro-info the shell utility [17:28] Jeroen has recently created the BootImage table so it sounds reasonable to use that. [17:29] Temporary as in get-andres-unblocked right now. [17:33] smoser: definitely not something that should go in 12.10. I'll talk to Jeroen about that tomorrow. But BootImage should be populated by maas-import-pxe-files which uses distro-info. === cmagina is now known as cmagina_away [17:49] smoser: it's trying to hit archive.ubuntu.com and not ports [17:52] rbasak, can i see logs ? [17:55] smoser: so at least two problems that I can see [17:55] It hits archive.ubuntu.com [17:55] and resolv.conf is still broken [17:55] smoser: I'm EOD+1h now [17:55] smoser: but if it'll help I can show you how to test on this machine yourself [17:56] It's pretty straightforward [17:56] rbasak, your'e not booting that image. [17:57] smoser: I just this moment realised that [17:57] sorry [17:59] smoser: so I only see one /var/lib/maas/...disk.img which I've replaced [17:59] rbasak, restart tgtd [17:59] and make usre you updated your v/ar/lib/maas/tftp/..... kernel and initrd.gz [17:59] sorry. i know you're edo [17:59]