[00:07] hatch you still around? should I bother with a review? [00:14] hatch gave you a quick LGTM with small comment anyway. :-) thx [00:14] night [00:14] all === schwuk_away is now known as schwuk === schwuk is now known as schwuk_away === schwuk_away is now known as schwuk === schwuk is now known as schwuk_away === schwuk_away is now known as schwuk [12:28] frankban: have you succesfully run the charm functional tests recently? I can't get past "agent-state: pending" [12:29] benji: no, not recently, maybe teknico run them for his last branch? [12:33] benji: are you using pyjuju? if so, you can try to run them with juju-core. what's the status of the juju-gui machine? [12:33] yep, I'm using pyjuju; I guess I could try that [12:35] frankban, does marco's test framework treat pyjuju as a first class supported environment, or is it second class, after juju core? [12:39] gary_poster: I believe they are both supported at the same level; juju-test automatically retrieves the version in use and reacts accordingly. [12:39] ok cool, thanks frankban [12:41] gary_poster, hazmat, teknico: I'll schedule a meeting in the calendar to talk about the deployer; does tomorrow at 1500 UTC work for you? [12:42] frankban, not for me. Thursdays are horrible for me :-) s'ok can meet without me and summarize if you want to [12:42] gary_poster: Friday would work as well, perhaps at 1400 UTC [12:43] frankban, that works well for me [12:44] oh frankban I am taking Friday off :-P [12:46] frankban, btw, and maybe relatedly, would you and teknico mind permanently moving our weekly calls one hour earlier? frankban's would be 1300 UTC and teknico's would be 1330 UTC. Is that lunch time? if it doesn't work, np [12:49] gary_poster: isn't our call at 1300 UTC already? anyway, no problem for me with moving the call one hour earlier. starting from tomorrow? [12:49] our call is currently 1400 UTC, if google can be trusted [12:51] heh, it's 15:00 – 15:30 in my calendar, and we are currently in CEST time (utc+2) [12:52] heh [12:52] gary_poster: our call is currently 16:00 local, 14:00 UTC. the new time would be half an hour earlier for me [12:53] gary_poster: it's not lunch time now, it will be once DST is off and local is back to UTC+1, in November, but it would still be ok for me [12:54] benji: I did repeatedly run "make ftest" on the charm successfully recently, but only with juju-core, I'm having problems bootstrapping with pyjuju [12:54] thanks for the info, teknico [12:54] teknico, frankban, I am utterly confused, particularly with what teknico said :-P, but will make the adjustments in Google calendar and hope they all work out all right :-) [12:55] gary_poster: sorry about that :-) [12:55] gary_poster: they will work, and I'll set up a possible deployer meeting after talking with you tomorrow [12:55] frankban, teknico, cool, thanks :-) [12:56] hey bac, it's 8:56 for you right now, right? [12:57] 8:57 [12:57] :) [12:57] :-P [12:57] :-) [12:58] fwiw, and you needn't bother remembering, PR is ATC which is always UTC-4 [12:58] AST [12:58] cool [12:58] thanks [12:58] +1 for consistency. [12:58] yeah [12:59] gary_poster: Google Calendar now says that our call tomorrow is at 14:30 local, which means 12:30 UTC, is that intended? if it is, it's ok for me [12:59] so, anyway, sorry was going to ask you to move meeting time but nm [12:59] my campaign: dump daylight savings time. buy all dairy farmers Tivos [12:59] bac: yeah, DST is madness [12:59] teknico, yeah. I think my issue is that Google says that my timezone is UTC -5, but they really mean I'm UTC-5+1DST [13:01] gary_poster: right, ok [13:01] benji! hi. you may be noticing a pattern here :-P . May I move your call to the morning, UTC 1300 (now but tomorrow)? [13:02] gary_poster: sure [13:02] thank you [13:02] gary_poster: is that perminent? [13:02] benji yes [13:02] k [13:02] still ok? [13:02] k [13:02] thanks [13:02] yep [13:02] cool [13:03] orangesquad. I will be distracted by charmworld's prodstack mongodb [13:03] sinzui: rgr [13:03] sinzui: ACK [13:04] thank you sinzui. if you want anyone from GUI to pair, lemme know [13:11] hatch: when you're around wonder if you have time to chat. [13:20] abentley, rick_h. production mongo shows evidence that mongodb has grown beyond our intent [13:21] sinzui: Do we have an idea what collections are eating the space? [13:21] hatch, Makyo, bcsaller I adjusted our one on ones permanently. My Thursdays were unpleasantly insane, plus I had a new conflict with Matt and Jeff's time. lemme know if these times are ok for you please [13:22] abentley, rick_h: there are two more dotted collections in production than we have in staging and the are 2 and 4 x larger than the collection we use: 513M juju.3, 1.1G juju.4 [13:22] * benji weeps softly into his keyboard. [13:22] sinzui: well that's interesting. :/ [13:22] Why is getting juju-core to run so hard. [13:22] heh [13:23] benji, dunno. we've got two resources to help ready now, at least, and Matt later [13:23] sinzui: Are those databases, not collections? [13:24] abentley, rick_h: I would like to rm them, but I know little of mongodb operations. [13:24] gary_poster: who would be the best person to ask? [13:24] abentley, right, db backups/something [13:24] benji, frankban probably. teknico and Makyo also have a working juju core installation. [13:25] sinzui: I don't know much about this either. So these are files, not db names? [13:25] abentley, nm, I see. mongodb scales its used from 64 to 128 to ... It has scaled up twice beyond staging [13:25] * benji waits for one to volunteer... before choosing a victim. [13:26] benji: what's the problem? :-) [13:26] abentley, they are named juju.x so I think they are the db, but I believe mongodb is preallocating storage in the files. [13:27] frankban: the problem is I don't know where these things are written down, but the immediate problem is "error: no tools available" [13:28] sinzui: have you tried to run a mongo db compact on the db? [13:28] rick_h, that takes the db offline (I thnk) [13:28] sinzui: I'm curious if it's pre-allocating and messing with things if a compact would correct [13:28] sinzui: true, can we test on staging to get an idea of the length of time? [13:28] rick_h, I think you are right [13:28] benji: that's a new one... does it occur running juju bootstrap --upload-tools? [13:28] sinzui: if we have to bring up a new instance we'd have to go offline as well. [13:29] frankban: nope, that's a bare bootstrap; I'll try with --upload-tools [13:29] sinzui: but this is something we've known we'd have to figure out at some point. I'm not sure if we can backup, bring up a new mongodb instance, and swap the front end over to it [13:29] rick_h, you mean bring up a new unit, compact the old. [13:29] sinzui: if we can, maybe we can get an idea of the size of the same data, but on a fresh mongo, and at least tell if this is a compaction issue vs a "you have too much data" issue [13:30] sinzui: on the newly brought up instance, while leaving things running, for now, on the old [13:30] actually, if we bring up a new unit, we will see if the juju db was given gigs instead of megs [13:30] sinzui: rgr [13:30] sinzui: that's my only thought atm. I don't know enough mongodb to help much beyond exploring things a bit. sorry [13:30] sinzui, rick_h: running compact will not delete any files and apparently temporarily increases disk space. [13:30] well we don't want to do that on this unit then [13:31] sinzui: right, but does backup/restore on a new instance shrink the size down to closer to what we expect? [13:31] abentley: :( [13:32] frankban: --upload-tools appears to have helped [13:33] sinzui: kapil seems very knowledgeable about mongodb disk usage. [13:33] benji: are you running juju-core from the compiled trunk? [13:34] frankban: yep (although I can't be sure it is up to date, as I am not sure I did the right thing to update it) [13:34] abentley, yes [13:34] frankban: if by "compiled trunk" you mean a trunk checkout that I compiled [13:35] I am going to query collection sizes and ask for the same from production to verify if the data is about the same [13:35] damn, this is a new unit. my mongo query history is gone [13:36] sinzui: This page has some queries: http://docs.mongodb.org/manual/faq/storage/ [13:37] abentley, I had queries in my history that iterated over our collections and gave me the counts [13:37] sinzui: That page has queries that iterate over collections and give you counts. [13:37] benji: yes, I was trying to understand that error. anyway, when you compile and run juju from trunk, it makes sense to always use --upload-tools [13:43] frankban: thanks for the help, I appear to have the tests running succesfully now. We'll know for sure when the first test finishes. How long do they normally take? (I [13:43] (I'm running them in EC2.) [13:43] 30min IIRC [13:44] benji: ^^ [13:52] yay! passing tests! Thanks much frankban. [13:54] great [13:56] I wonder if pyjuju is having intermittent machine provisioning problems. [14:00] luca__, hey, guichat? [14:01] gary_poster: give me 2 mins! :D [14:01] luca__, cool :-) [14:06] morning [14:15] rick_h: do you have a moment to chat? [14:16] jcsackett: sure thing [14:50] brb [14:55] benji, bac: Remember to run "make lint" before landing on charmworld. I just fixed a bunch. [14:55] orangesquad: could one of your review this mp: https://code.launchpad.net/~adeuring/charmworld/1199790-last-change-error/+merge/177865 ? [14:55] abentley: I'll certainly try. We should add that to the CI. [14:55] adeuring: looking. [14:56] abentley: will do. thanks. [15:00] bak [15:00] benji: I am hesitant to add something to CI that will prevent working code from landing. [15:01] benji: For example, we lived for a long time with a lint error because one file was doing "from x import *". [15:01] It seems to me that the only way to get something we want is to enforce it. Automatic enforcemnt would appear cheaper. [15:03] benji: I think social pressure is sufficient here and avoids the inflexibility of automatic enforcement. [15:05] It comes down to how we weigh the costs, benefits, and probabilities. I weight the cost of someone having to resubmit a branch because of a lint error low (partially because I would propose a make target that would lint and propose), the cost of someone else having to deal with my lint errors high, and the probability of "trying harder" to materially increase compliance as low. I can understand others having different weights/probab [15:06] also the possibility of a lint error causing an obscure issue...... +1 benji....sorry abentley :) [15:06] luca__: you're back! wb [15:07] benji: How about we start with the make target? [15:08] sounds good to me [15:13] sinzui: The Kanban card "Implement new primary key for charms" should not have gone into Archive. It's unlanded work (that I'm about to land). [15:13] abentley, yes, but oh [15:14] I believed you were done with that card, you were going to land the branch merged into another branch...tracking the other card all the way to the end of the board [15:14] rick_h: sorry I missed in the backscroll that you wanted to chat? [15:15] hatch: yea, guichat? [15:15] ypup there [15:17] sinzui: Oh. I thought we would handle that situation by moving the card when the follow-on branch landed. [15:18] abentley, I had planned that originally, but since we were vacating the board, I decided to treat the issue as "the work is done because any developer can build of the branch". [15:19] abentley, sorry, you were off when I made the decision. I talked to everyone by you. [15:19] I suck [15:19] sinzui: Nah, I understand what happened now. [15:19] sinzui: All good. [15:20] sinzui: No need for a 1x1 today, right? [15:21] abentley, +1 [15:21] hey hatch, thanks :) [15:43] jujugui: can i get two reviews on https://codereview.appspot.com/12029045 ? [15:43] jcsackett: I'll do one. [15:43] benji: thanks! [15:43] jcsackett: yep [15:43] bac: thanks. :-) [15:52] jujugui calli n 8 kanban now [15:58] jcsackett: done but i haven't qa'd yet [15:59] jujugui call in 1 === schwuk is now known as schwuk_away === schwuk_away is now known as schwuk [16:01] jcsackett, Starting now [16:16] gary_poster: comingsoon is 13 versions behind trunk [16:18] mongodb people, this is a summary of what I know about manage.jujucharm.com's db https://pastebin.canonical.com/95291/ [16:22] no one be alarmed if you heard a crashing noise as i signed off. my dog just discovered the side table is rickety and falls over if you nose it. :-P [16:22] lol [16:23] sinzui: Our canonistack instances have /dev/vdb mounted at /mnt, giving us 10 GB of space. Are the prodstack instances similar? [16:25] gary_poster, hatch: comingsoon is now r914. the cronjob step 'make build-prod' was failing due to the switch to the new version of d3. it needed a 'make clean' to get the cruft out. [16:25] ahh great thanks bac [16:28] hey luca__ [16:29] abentley, I am asking. I know production has 10G but logs are on a another disk [16:29] hatch: heya [16:30] bac, awesome thanks [16:31] jcsackett, if you want me somewhere I'll come by, otherwise will go back to planning world domination, or something [16:31] gary_poster: i think we're good for now. [16:31] cool jcastro [16:31] wrong name, dude. :-P [16:31] sorry tab completio n :-P [16:31] cool jcsackett :-P [16:33] crap [16:33] guess I'm taking my engineer-license again [16:33] :P [16:34] gary_poster: if you change something and then hit the toggle it'll reset the values and that's what they will stay [16:34] oh he's back [16:34] and he's gone [16:34] lol [16:36] hatch, +1 [16:44] debug hooks in juju core: https://codereview.appspot.com/12019043/ [16:45] someone working on core for 2 weeks :-) [16:56] check out this css rule `input#use-default-toggle:checked ~ label .handle` [16:56] OOOOOeeeeee [17:21] jujugui, I moved my notes to a gdoc so that we can all comment and edit. https://docs.google.com/a/canonical.com/document/d/1AoMhTdrominYLBS8iHbZLW_Ji2OH3mNhUzoHjetnD9w/edit# === schwuk is now known as schwuk_away [17:49] jcsackett: got a sec for a chat? [18:12] gary_poster: I think this is completed https://bugs.launchpad.net/juju-gui/+bug/1201023 [18:12] <_mup_> Bug #1201023: Charmbrowser sidebar elements all have scroll bars in IE10

[18:14] rick_h: sure. was grabbing lunch. [18:14] rick_h: still need to chat? [18:14] jcsackett: no, I'm ok now. I follow what's up. Got confused by the jujucharms browser view setup [18:15] rick_h: ah, yeah. i could see that. [18:15] rick_h: i *think* maybe we can kill that? since we've launched, and i've heard nothing about doing a proper landing. [18:15] jcsackett: yea, I was tempted to but I can work around it for now [18:15] was the basis of the conversation request [18:17] rick_h: dig. [18:51] ugh handlebars drives me friggen bonkers sometimes [18:56] rick_h: you might be able to help me out here [18:56] hatch: what's up? [18:56] service-configuration.partial has a {{#settings}} loop [18:56] and within that loop I need to access a parent property [18:56] ../ [18:56] I tried ../ghost (ghost being the property) [18:56] but no luck [18:56] {{#if ../ghost}}disabled{{/if}} [18:58] hmm, app/templates/unit.handlebars uses it [18:58] are you sure that there's a ghost one scope level up? [18:58] yup, logged it out [18:58] heh {{debugger}} ftw? [19:01] hmm, I wonder if it's because of the scope block of an #if [19:01] yeah checking that now [19:01] I upgraded the debugger fn [19:02] ok the #if must have busted scoping? [19:02] to #YUI! [19:03] hatch: so yea, I'd drop a debugger in the YUI handlebars code in the if block helper and see wtf 'this' is in there. I'm betting it's not the loop scope any more? [19:03] hatch: otherwise, I'd go about it a different way and try to get the var into the loop scope. Sucks, but that's why we have some buildTemplateData type helpers lying around. [19:03] if that was the case then `ghost` would work, but it doesn't either [19:04] yea, the only way to tell is to break in the if block helper and look [19:04] http://stackoverflow.com/questions/11562612/handlebar-context-lost-after-passing-handler [19:04] I don't recall using a block with a ../ up scope variable reference. [19:05] hatch: yea, that makes sense [19:05] so does ../../ghost help you out? [19:05] nope, maybe I need three [19:05] or ../../this.ghost [19:05] yup three [19:05] but yea, basically a bunch of wheee to figure out how to get out of the context in the block helper [19:05] ugh... [19:05] yea [19:06] handlebars totally does not follow the path of least astonishment [19:06] haha [19:06] well, you're in the base context, then the loop context, then the if context [19:06] then the wtf am I doing context [19:06] yea, I'm a 'give me my code or give me death' camp on templates myself. Unfortunately Mako hasn't been ported to JS yet :) [19:06] lol [19:06] time for a doc to explain wtf ../../../ is for :P [19:07] :) thanks for the help [19:07] lol [19:07] hah, np. "Keep adding ../ until it works darn it!" coaching is here for you [19:07] hahaha [19:11] ugh, that damn showView made some stuff magically work. [19:15] rick_h: so depending on how many if blocks your in, you need more or less ../'s [19:15] oy vey [19:15] hatch: but of course, each is another context [19:15] you need good tests on those. If someone moves the if blocks around then things will go nuts [19:16] hatch: but if it's getting too crazy maybe it's time to rethink? [19:20] rick_h: Yeah I think I'll switch it to a helper [19:20] but helpers don't have access to the full stack though I don't think [19:20] so maybe just some comments [19:24] lunching [20:28] hey bac, you actually here, or juts pretending to be? I've got wedgwood in webops saying that he wants to rip out some of our old stuff from charm-helpers [20:28] my memory is that we tried to get some stuff approved [20:28] hi gary_poster [20:29] but he didn't like test helpers or something like that [20:29] hey [20:29] you don't have to be here if it is past EoD :-) [20:29] yeah, he didn't promote the test helper stuff, saying it was client code [20:29] gary_poster: no, it is just 16:30 [20:29] same as you [20:30] yeah I know but don't know when your start time is anymore :-P [20:30] gary_poster: i've been starting at 8 [20:30] cool [20:30] gary_poster: anyway he wants to remove the non-promoted stuff? [20:30] bac, yeah [20:31] we still are not using it anyway, right? I didn't see it anywhere [20:31] i think that code is useful and deserves to live some place [20:31] gary_poster: huh, i thought our tests used it [20:31] gary_poster: or was it just from buildbot? [20:32] bac no I meant we are not using wedgwood's library yet [20:32] gary_poster: that is true [20:32] bcsaller: can you think of a way to reset all of the settings fields to their defaults using current code? I was thinking something in the databinding maybe to reset to the default model values? [20:32] gary_poster: if it is the OneTrueWay we'll need to spend some time converting our charm [20:33] but that is tech debt for us now, so i'm not sure when we'll carve out the time [20:33] hatch: off the top of my head, viewlet.model.set(viewlet.model.getAttrs()) [20:34] oh, defaults [20:34] oo good idea lemme try that [20:34] not prev value though [20:34] defaults come from the charm, not the values in the service [20:34] model.set('config', charm.options) [20:35] should work though [20:36] rick_h: why would 'make testid' produce non-contiguous numbers? i'm seeing things like 1..16, 420, 17, 18, 421 [20:40] bcsaller: ahh actually we aren't databinding the ghost config so that approach wont' work :( [20:40] will need to just loop through I guess, oh well [20:40] hatch: I thought we changed it to db everything now though? [20:41] ohh hmm lemme look further maybe I broke something [20:45] bcsaller: do I have something wrong here https://gist.github.com/hatched/beee9956434d9ff6be5f the update fn is never being called when I update options [20:47] hatch: you're sure its options and not config now, I'm not sure the current state of that [20:47] but the structure looks right [20:48] well regardless of what it is - changing that object 'should' fire that update method [20:48] no, update in bindings is called when that binding changes [20:48] so if the name is wrong it doesn't happen [20:48] ok but there is an attribute called 'options' [20:49] which I am setting [20:52] we have options, config, settings, depending on which model you have I'd guess the name was wrong but I don't know how far you've gotten tracking it down [20:55] I'm just going to loop through the default values [20:55] I think this is a bug, I'll document it later and see if we can resolve sometime [20:56] is the model is question a POJO? it might be that observe isn't triggering change on a nested complex object [20:56] ohh wait a second [20:56] I think this is a bug from the charm model stuff [20:57] charm model changeover [20:57] all of the fields are undefined [20:57] * hatch shakes fist at jcsackett [20:57] lol :P [20:57] hatch: i qa'ed against service inspector and didn't see anything. [20:58] options.options seems weird, if options is charm...the charm attr should be get('options') [20:59] bac: not sure, just how it works. I've not looked at the implementation [20:59] bac: the test runner usually loads tests and runs them alphabetically and such. SO maybe something with loading/running [20:59] bcsaller: jcsackett ahh I found the bug....the fields have been renamed in the charm model and now the template bings on the wrong object [20:59] easy overlook [21:00] * bcsaller coughs [21:00] thats what I said [21:01] yeah well I had no idea what you were trying to say [21:01] * hatch blames internet [21:01] lol [21:02] * bcsaller blames unicode [21:02] * hatch blames timezones [21:03] (all developers can agree timezones are to blame so it's an easy scapegoat) [21:03] yeah, we should get rid of those [21:04] aww man I got up so early. Yeah what time? oh like 5pm [21:04] hehe [21:07] hatch: which fields were renamed? i was trying to avoid doing that where possible and am concerned something may have been landed that shouldn't. [21:08] jcsackett: it's no big deal it's just a discrepency between the ghost and service models and their resulting structure [21:08] I think it's done properly now so I'm taking another approach [21:08] I was just blaming you because your current task sucks [21:08] ;) [21:08] hatch: :-P [21:09] :-) [21:09] go jcsackett! sis boom bah! :-) [21:20] sinzui, how goes the MongoDB war? [21:21] webops are considering my recommendations to move the db files to a larger mounted partition. This suggest means 0 down time [21:33] bcsaller: so I don't think that setting an object to the same value as it is triggers the databinding [21:35] it's ok, I'm looping through the fields [21:36] but just thought I'd point that out [21:40] sinzui, cool. do we know why the data is more than we expected? [21:41] mongodb perallocates ever increasing files when there is lots of activity [21:41] gary_poster, This is the summary: https://docs.google.com/a/canonical.com/document/d/1AoMhTdrominYLBS8iHbZLW_Ji2OH3mNhUzoHjetnD9w/edit# [21:43] sinzui, ugh. thank you. [21:44] Jeff Pihach has been assigned to the Card [(1201024) IE 10 drag and drop bug] by Gary Poster. NOOOOOOooOOOOoooooooooo [21:44] * hatch curls up under the desk [21:44] :D [21:44] hatch lol, that was the one you fixed before [21:44] hatch, I moved that to archive and gave you credit [21:45] there is still the new one :-) [21:45] ohh haha ok well then that's better :D [21:45] which I have not assigned to you...yet... [21:45] lol [21:45] who displeases me, among you?!! mwa ha ha ha. etc. [21:48] hahah [21:50] benji, where is the charm branch? [21:50] was hoping we could have that ready today [21:51] by "where is" I mean "what's going on with the." By "benji" I mean "benji, who is past EoD," [22:03] :D [22:03] gary_poster: have a second to try out my toggle branch? lp:~hatch/juju-gui/use-def-config [22:04] hatch, I think so, unless pulled away suddenly :-P one sec [22:07] cool thanks! [22:11] hatch, pulled away :-P will look asap after [22:12] haha np [22:22] gary_poster: https://code.launchpad.net/~benji/charms/precise/juju-gui/fix-cache-headers/+merge/177958 [22:22] darn, I'm in the right channel [22:22] * benji goes to a channel far, far away [22:35] hatch, (new Array(3)).map(function(d) { return 0; }) === [undefined, undefined, undefined]; // what's happening here? [22:35] Or anyone, really. [22:36] magic! [22:36] wait [22:36] that's gota be d3 code [22:36] because it's impossible to read [22:36] I think I know what's happening [22:36] hatch, From a friend, just asking around. [22:36] it creates a new array with 3 empty slots [22:37] Just curious because [undefined, undefined, undefined].map(function(a) { return 0; }) === [0, 0, 0] [22:37] it then loops through each one returning 0 [22:37] so each one will be [0,0,0] [22:37] Makyo: it's probably some wako way to test for map functionality [22:38] or proper functioning map functionality [22:38] The result should be [0, 0, 0] since (new Array(3)) === [undefined, undefined, undefined] [22:38] I would love to know the real reason behind the code [22:38] hatch, not from an open source project, friend works elsewhere. [22:38] right, but if map wasn't working properly then it 'could' be an indicator [22:38] So I'm assuming there's some more intermediary stuff, probably conditional. [22:39] right....still odd [22:39] Well, those results are from my console. [22:40] Oh, huh. (new Array(3)).map(function(a) { return typeof a; }) === [undefined, undefined, undefined] [22:40] but [undefined, undefined, undefined].map(function(a) { return typeof a; }) === ["undefined", "undefined", "undefined"] [22:40] So the constructor is just sort of...doubly undefined :P [22:41] [undefined, undefined, undefined].map(function(d) { return 0; }) [22:41] gives what one would expect [22:41] returns [0, 0, 0] [22:41] right [22:41] as it should [22:41] my guess is this is some edge case from using the array constructor [22:41] which they say never to use :) [22:41] Hahaha [22:41] Yeah [22:42] I'm intrigued now [22:42] Ohhh, he found it: http://stackoverflow.com/questions/5501581/javascript-new-arrayn-and-array-prototype-map-weirdness [22:43] ahhhhh [22:43] There's a difference between undefined pointers and pointers to undefined. [22:43]