| nacc | rbasak: https://paste.ubuntu.com/p/M4nvkWnD6n/ | 03:38 |
|---|---|---|
| nacc | powersj: https://code.launchpad.net/~nacc/usd-importer/+git/usd-importer/+merge/336877 please | 03:39 |
| nacc | it should still fail, but with the same three errors as the above paste | 03:39 |
| nacc | rbasak: you had mentioned before using dpkg-deb or something? | 05:15 |
| === MJCDoffice is now known as MJCDoffice1 | ||
| === MJCDoffice1 is now known as MJCDoffice2 | ||
| === MJCDoffice2 is now known as MJCDoffice3 | ||
| === MJCDoffice3 is now known as MJCD_ULTIM8_ | ||
| === MJCD_ULTIM8_ is now known as MJCDoffice | ||
| cpaelzer | good morning | 06:33 |
| hallyn | kirkland: does the new ubuntu server installer still work with preseeding? | 06:46 |
| lordievader | Good morning | 07:04 |
| frickler | jamespage: coreycb: can you trigger https://bugs.launchpad.net/designate-dashboard/+bug/1715417 to be pulled from artful into uca/pike, please? | 09:31 |
| ubottu | Launchpad bug 1715417 in Ubuntu Cloud Archive pike "Cannot view a zone in dashboard - 404 errors" [Medium,Fix committed] | 09:31 |
| jamespage | frickler: can do as soon as someone marks the version in uca pike/proposed as verified :-) | 09:33 |
| rbasak | nacc: try this: http://paste.ubuntu.com/p/7JgxcjV5w6/ | 09:36 |
| rbasak | nacc: it stops building the .changes file. We could use dpkg-genchanges directly if we need it. | 09:36 |
| rbasak | nacc: and if there are any differences, well all tests still pass, so we can deal with that if and when we find it's insufficient. | 09:37 |
| frickler | jamespage: ah, I missed that tag, will do in a bit, thx | 09:56 |
| === _Jeepbeats is now known as Jeepbeats | ||
| === Screedo_ is now known as Screedo | ||
| === techmagus_ is now known as techmagus | ||
| === led_ir23 is now known as led_ir22 | ||
| === lynxman_ is now known as lynxman | ||
| === rmk` is now known as rmk | ||
| === DalekSec_ is now known as DalekSec | ||
| jamespage | coreycb: ok I've done a few more tweaks on xtrabackup and imported and pushed the resulting work back to the main percona-xtrabackup repository | 12:00 |
| jamespage | coreycb: most of that time was a copyright audit :-) | 12:00 |
| jamespage | coreycb: all building in https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3160 | 12:00 |
| jamespage | coreycb: I also did the same watch file and repack exclusions as I did for pxc-57 | 12:02 |
| jamespage | limites the size further and helps with a load of compressed js inclusiosn | 12:02 |
| jamespage | frickler: well ceph 12.2.3 was an experience | 12:06 |
| jamespage | frickler: I was not expecting a boost version bump in a point release | 12:06 |
| jamespage | frickler: thanks for testing the pike proposed designate dashboard - promoting that now | 12:09 |
| cpaelzer | jamespage: coreycb: I was asked if/why pike seems to be out of date | 12:27 |
| cpaelzer | jamespage: coreycb: plenty of fixes in artful up to https://launchpad.net/ubuntu/+source/qemu/1:2.10+dfsg-0ubuntu3.5 are not seen in there | 12:27 |
| cpaelzer | is this in staging? | 12:27 |
| frickler | jamespage: I think the main misunderstanding about ceph is that they do not use semantic versioning, but their version numbers look like they did | 12:28 |
| cpaelzer | yeah see it in staging | 12:28 |
| cpaelzer | any ETA for when those are released that I could share? | 12:28 |
| jamespage | frickler: that's a change in behaviour then - to-date point and patch releases have been just that | 13:04 |
| jamespage | cpaelzer: I'd like to get those clear today but need to laise with coreycb first | 13:05 |
| coreycb | jamespage: cpaelzer: ok pike and ocata have been regression tested successfully as of 2/15 | 13:09 |
| coreycb | jamespage: cpaelzer: pike should be ready to promote. it's already promoted for artful and regression tested. i want to do some more thorough testing for kilo/ocata though as i had to adjust patches to backport those. | 13:10 |
| jamespage | coreycb: ok actioning pike now | 13:11 |
| jamespage | coreycb: actually can you annotate the bug for the point releases with test results and updates the tags :-) | 13:15 |
| jamespage | coreycb: btw https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3172/+packages is working good for me on a three unit bionic cluster | 13:16 |
| frickler | jamespage: adding a new binary in 12.2.2 and severely changing default parameters already wasn't too nice either | 13:17 |
| coreycb | jamespage: all set bug1744882 | 13:17 |
| coreycb | bug 1744882 | 13:17 |
| ubottu | bug 1744882 in qemu (Ubuntu Bionic) "Add SPEC_CTRL and IBRS changes" [Undecided,Triaged] https://launchpad.net/bugs/1744882 | 13:17 |
| jamespage | ack done | 13:18 |
| coreycb | jamespage: \o/ 5.7 | 13:18 |
| frickler | jamespage: iiuc the issue is that ceph calls every update to luminous 12.2.x, while 12.2.2 should really have been 12.3.0 and 12.2.3 == 12.4.0. there haven't been so drastic changes to jewel I think. but maybe they'll learn for 13.* | 13:22 |
| xnox | smoser, i see why the package is not there =) it got dropped to universe. | 13:25 |
| xnox | and ubuntu-meta builds with main only, asking for it to be promoted back into existance. | 13:27 |
| kirkland | hallyn: actually, it uses curtin and cloud-init! | 13:33 |
| kirkland | hallyn: which makes it basically the same as MAAS | 13:34 |
| tobasco | jamespage: looks like it's still py3 in xenial-proposed/queens | 13:51 |
| jamespage | erm | 13:55 |
| jamespage | tobasco: http://paste.ubuntu.com/p/sdHSXmrfCP/ exists; the default is still py3, but its possible to use py2 via that package - it contains all of the required binaries and wsgi entry points | 14:05 |
| jamespage | coreycb: doing some sysbench against those pkgs - hanging together ok so far | 14:13 |
| coreycb | jamespage: awesome | 14:16 |
| jamespage | coreycb: ok upload those both to bionic | 14:20 |
| jamespage | coreycb: good work btw - a bit of a team effort which is good | 14:21 |
| jamespage | as we both now know :-) | 14:21 |
| coreycb | jamespage: woohoo \o/ thanks for all the help! | 14:21 |
| smoser | xnox: bah. what do we need to do then ? | 14:39 |
| tobasco | jamespage: ah ty | 14:48 |
| xnox | smoser, asked archive admin to repromote it back in. it is done and now waiting to publish. | 14:52 |
| xnox | it's a "binary only movement to main" | 14:52 |
| smoser | xnox: someone must have done it. | 14:53 |
| smoser | http://paste.ubuntu.com/p/DJdPPxhzmW/ | 14:53 |
| smoser | uploading. | 14:53 |
| smoser | cpaelzer: around ? | 14:55 |
| smoser | can we discuss bug 1750780 | 14:55 |
| ubottu | bug 1750780 in open-vm-tools (Debian) "Race with local file systems can make open-vm-tools fail to start" [Unknown,Incomplete] https://launchpad.net/bugs/1750780 | 14:55 |
| xnox | smoser, yeah =) it was not there like 40mins ago, so it must have just published. cool. | 14:56 |
| xnox | smoser, once that publishes and migrates; i guess we need to respin images? then retry the tests? then things will migrate? | 14:56 |
| smoser | i uploaded open-iscsi with the timeout yesterday, xnox | 14:58 |
| smoser | i'mok if you let stuff through. it really is at this pomoment "bad test" | 14:58 |
| jamespage | coreycb: OK I've sniffed queens in proposed sufficiently - promoting to -updates | 15:03 |
| coreycb | jamespage: ok | 15:03 |
| xnox | smoser, ok, proposing a hint https://code.launchpad.net/~xnox/britney/open-iscsi-vs-broken-cloud-image/+merge/338560 | 15:10 |
| smoser | xnox: can you just suggest that it should be removed if there is a cloud image > 2018-02-22 | 15:12 |
| smoser | or more specifically with 'server' at 1.411 ? | 15:13 |
| smoser | err.. ubuntu-server | 15:14 |
| xnox | smoser, that's not enough, we need image with server at 1.411 which is hard to codify =) the hints are manual.... | 15:14 |
| smoser | sure. i'm saying write that in text | 15:14 |
| xnox | smoser, however, the hint will self-become inactive, once the open-iscsi package in proposed passes the test and migrates. | 15:14 |
| xnox | smoser, so in effect, it will self-drop automatically when everything becomes good. | 15:15 |
| smoser | oh it will? | 15:15 |
| xnox | smoser, note the hint is version specific. | 15:15 |
| smoser | i didnt know there ws any magic | 15:15 |
| xnox | it's not an /all/ hint | 15:15 |
| smoser | ah. you have the version of open-iscsi i see. | 15:15 |
| smoser | at least stil write in text | 15:15 |
| smoser | if image has > ubuntu-server 1.411 | 15:15 |
| smoser | then this is not valid. | 15:15 |
| irvingwashington | anyone here know who to talk to about AMIs in cn-north-1? None of recent 16.04LTS images in cn-north-1 are bootable | 15:29 |
| smoser | Odd_Bloke: ^ | 15:34 |
| philroche | irvingwashington: smoser: This is unexpected. We will take a look at this. | 15:38 |
| Odd_Bloke | irvingwashington: Which AMIs have you tried? (Do older ones work for you?) | 15:39 |
| hallyn | kirkland: so I can't use preseed? | 15:45 |
| hallyn | kirkland: let me put it more construcively - would love to see a blog post about how to direct it to setup a particular paritioning scheme :) | 15:45 |
| hallyn | (and maybe how to then do some setup afterward using cloud-init - that part is pretty clear to me but still woudl be good for a blog post) | 15:46 |
| kirkland | hallyn: indeed, that's a good question | 15:46 |
| hallyn | BTW there's another blog post / tutorial which would be good to see - how to write a simple replacement set of MAAS scripts to control the power controllers | 15:47 |
| hallyn | No such post out there right now, would be both useful and probably helpful to maas adoption | 15:48 |
| cpaelzer | thanks for the inof coreycb and jamespage | 15:51 |
| smoser | hallyn: well, "scripts" dont work. you can plug in a power controller to maas, but its integrated. python. | 15:51 |
| smoser | not "scripts". | 15:51 |
| cpaelzer | smoser: I'm here | 15:52 |
| cpaelzer | smoser: in your standup hangout now if you want? | 15:52 |
| smoser | k | 15:53 |
| smoser | hallyn: https://gist.github.com/smoser/375123ef1ef098be23cc856a5772c5c8 | 15:53 |
| smoser | that describes how you can kind of test things and such. | 15:53 |
| smoser | http://curtin.readthedocs.io/en/latest/ | 15:53 |
| coreycb | jamespage: i'm going to try building pytest without the pypy-hypothesis BD and possibly patch that in ca-patches | 15:53 |
| smoser | has information on curtin, the 'configuration types' | 15:53 |
| smoser | are how to do storage layouts | 15:54 |
| smoser | and then... there are many examples in tree of storage. | 15:54 |
| smoser | hope that helps. | 15:54 |
| cpaelzer | smoser: ah I found your bug update - reading ... | 15:54 |
| hallyn | smoser: yeah yeah everyone keeps 'correcting' me on that - if it's interpreted it's a script in my lexicon :) | 15:57 |
| hallyn | smoser: cool gist - you should make it a blog post :) | 15:59 |
| Odd_Bloke | irvingwashington: I can reproduce the issue with the HVM instance-store AMI (ami-fc459891), but not with the HVM EBS AMI (ami-cc4499a1). | 16:00 |
| Odd_Bloke | I'll dig in to the instance-store issue. | 16:00 |
| hallyn | smoser: so the ubuntu installer is now based on curtin; if i do a pxe boot of a netboot image of bionic ubuntu server, can that curtin config file be a url? | 16:01 |
| nacc | rbasak: are you ok if I pull that into my branch? | 16:01 |
| smoser | hallyn: yes i should | 16:02 |
| hallyn | I assume if I did it 'by hand' i woudl boot a liveos and run curtin from there - that's fine, but presumably the installer ... | 16:02 |
| coreycb | jamespage: yeah no dice on dropping pypy-hypothesis. maybe we don't need to backport pytest. | 16:04 |
| coreycb | checking | 16:04 |
| smoser | hallyn: *an* ubuntu installer is based on curtin | 16:05 |
| smoser | subiquity | 16:06 |
| smoser | which will be the primary offering for server | 16:06 |
| smoser | you will still be able to get the alternate download. | 16:06 |
| hallyn | ah. | 16:08 |
| hallyn | still - will subiquity be able to take a url for curtin config? | 16:08 |
| hallyn | i *do* want to be able to use curtin :) | 16:08 |
| smoser | i do not know that. | 16:09 |
| hallyn | ok :) | 16:09 |
| smoser | did you reply to that thread | 16:09 |
| hallyn | what thread? | 16:09 |
| hallyn | I need to try out mailborder, if that can do a decent job with my spam i'll try and un-devnull my ubuntu mail | 16:10 |
| rbasak | nacc: go for it | 16:11 |
| hallyn | smoser: anyway don't take it the wrong way but i'm gonna try and push you guys to blog a bit more :) | 16:11 |
| nacc | rbasak: thanks, building the snap locally | 16:11 |
| hallyn | I'm hoping in the next few weeks to play with the maas scripts (!) a bit and theni can blog about hwo to do it, | 16:12 |
| hallyn | but as i got pxe doing what i needed it's hard to justify the time | 16:12 |
| hallyn | :) | 16:12 |
| rbasak | nacc: I sort of intended you did that, as I can't easily test inside your snap environment, and didn't want to go further without checking that it actually does solve the in-snap problem. | 16:13 |
| nacc | rbasak: ack, understood :) | 16:13 |
| nacc | rbasak: can you also peek at https://git.launchpad.net/~nacc/usd-importer/commit/?id=3e6589aa6d2e4f53a5e76ecdb8fed2f12184f5e3 | 16:13 |
| nacc | rbasak: it's another obvious (to test in the snap, we need to adjust paths) | 16:13 |
| nacc | rbasak: the tests still work locally, as well, but i want to know if there's a better way to do that massage | 16:13 |
| smoser | hallyn: https://gist.github.com/smoser/9f9a2f521e13f3add8d45de00124c18d is related also | 16:14 |
| smoser | hallyn: yes. agree. | 16:14 |
| rbasak | nacc: I need to look up the context. But immediate thought: maybe wrap Changelog.from_path in the tests? | 16:14 |
| powersj | nacc: https://jenkins.ubuntu.com/server/job/git-ubuntu-ci-redux/7/console | 16:17 |
| nacc | powersj: thanks | 16:17 |
| hallyn | smoser: so i see i need to start watching https://gist.github.com/smoser :) | 16:18 |
| hallyn | can i get an rss feed of those i wonder | 16:18 |
| hallyn | smoser: cool looks like i shoudl update my ages-old uvt-kvm setup on my big host | 16:21 |
| irvingwashington | Odd_Bloke: we started with ami-fc459891 and worked our way backwards. Eventually gave up and went back to using the original 16.04 AMI we started from sometime in June of 2017 | 16:23 |
| Odd_Bloke | irvingwashington: All of them HVM/instance-store? | 16:23 |
| irvingwashington | Odd_Bloke: yes, we didn't test any ebs AMIs | 16:24 |
| smoser | hallyn: yeah. you should. :) | 16:24 |
| smoser | i do mean to take a bunch of those and turn them into blogs. | 16:24 |
| smoser | on github/blogs or whatever tha tis | 16:25 |
| Odd_Bloke | irvingwashington: OK, thanks for reporting the problem. :) | 16:25 |
| irvingwashington | we track 14.04 more closely and the recent images for that are fine. Only 16.04 | 16:25 |
| smoser | smoser.github.io | 16:25 |
| irvingwashington | Odd_Bloke: thanks for looking into this. Apologies for not reporting this sooner. | 16:25 |
| rbasak | nacc: http://paste.ubuntu.com/p/XtFrSQpbnZ/ | 16:27 |
| hallyn | smoser: i thought you'd run your own static site fed by m4. | 16:31 |
| hallyn | disappointed | 16:31 |
| nacc | rbasak: http://paste.ubuntu.com/p/QydVfYxr6b/ | 16:50 |
| nacc | rbasak: tests still fail, but differently | 16:50 |
| rbasak | nacc: I think that's a real bug. | 17:05 |
| rbasak | nacc: the patch I gave you worked locally, so perhaps a newer dpkg is more pedantic or something. | 17:06 |
| rbasak | nacc: the test needs updating to supply a version of '1-1' in the non-native case, instead of using the default '1' I think. Alternatively I need to fix SourceSpec or SourceFiles so if native is False then the version defaults to '1-1' instead of '1'. | 17:06 |
| nacc | rbasak: ok, i can look at it | 17:14 |
| === Epx998- is now known as Epx998 | ||
| nacc | rbasak: stil there? had a quick question | 17:48 |
| jair | hello there just to confirm, 17.10 is not a version of ubuntu I should be installing in servers right? like Dell R440 | 18:20 |
| dpb1 | jair: unless you are wanting to preview 18.04 features, I would not | 18:21 |
| jair | dpb1: understand | 18:21 |
| jair | dpb1: we installed in a dell r440 because the perc 740p raid controller in 16.04 server was not supporting it | 18:21 |
| dpb1 | jair: you could try the HWE kernel | 18:22 |
| jair | but now we have this weird issue the memory keeps growing every 19 hours and then the server crash | 18:22 |
| jair | Yep I got that tip but I am just trying to avoid re-install | 18:22 |
| dpb1 | jair: https://wiki.ubuntu.com/Kernel/LTSEnablementStack | 18:22 |
| dpb1 | ok | 18:22 |
| dpb1 | well | 18:22 |
| jair | dpb1: we are having this issue > https://ibb.co/dmrfvx | 18:23 |
| nacc | jair: please don't crosspost | 18:23 |
| nacc | jair: i was already helping you in #ubuntu | 18:23 |
| teward | jair: it *sounds* like you have rogue processes, rather than a hardware problem, stock Ubuntu on its own is not going to consume that much memory 'every 19 hours', more likely something you've got running is trying to take that memory | 18:23 |
| nacc | teward: it's THP, i'm fairly sure | 18:23 |
| jair | here is the output of meminfo > http://paste.debian.net/1011521 | 18:23 |
| nacc | teward: it's possible it's a rogue process doing the THP, but seems unlikely | 18:23 |
| teward | nacc: THP == ? | 18:24 |
| teward | i'm tired and uncaffeinated today :) | 18:24 |
| nacc | teward: Transparent Huge Pages | 18:24 |
| teward | thank you | 18:24 |
| * dpb1 backs away | 18:24 | |
| nacc | teward: they have a ton of memory allocated there | 18:24 |
| teward | you're right though THP is likely to be the problem | 18:24 |
| jair | nacc: my sincere apologies, I just noticed that ubuntu-server is where I should have been chatting | 18:24 |
| nacc | 1G pages, which i believe are not swappable | 18:24 |
| dpb1 | jair: ya, if you are already talking to nacc, you should just keep doing that | 18:24 |
| teward | whether it be rogue processes or not (but i just got here) | 18:24 |
| teward | dpb1: i presume you backed away because of the crosspost reason... or was it because I'm not caffeinated :P | 18:24 |
| jair | sorry all this is a server not a desktop | 18:24 |
| dpb1 | teward: lol | 18:25 |
| jair | therefore I believe I should be chatting here the Debian team advise me that | 18:25 |
| teward | jair: is this Ubuntu or Debian? | 18:25 |
| teward | there *is* a difference | 18:25 |
| teward | (Just confirming) | 18:25 |
| jair | Ubuntu | 18:25 |
| jair | 17.10 | 18:25 |
| jair | teward: here http://paste.debian.net/1011531 | 18:25 |
| teward | nacc: and we confirmed THP is enabled on their environment? | 18:26 |
| jair | I already did what nacc told me about disabling THP | 18:26 |
| teward | whoops speaking of memory issues... *grabs another stick of RAM to throw in the hypervisor that is almost out of RAM, disappears for a short while* | 18:26 |
| nacc | teward: yeah, madvise | 18:26 |
| jair | teward: it was http://paste.debian.net/1011527 | 18:26 |
| nacc | teward: i'm going off of their meminfo | 18:26 |
| jair | teward: I disabled it | 18:26 |
| nacc | jair: did you reboot yet? | 18:26 |
| jair | nacc: the server is still running and the memory increasing | 18:27 |
| jair | I suspect it will crash soon | 18:27 |
| nacc | jair: right, so reboot? | 18:27 |
| nacc | jair: not sure why that's relevant? | 18:27 |
| nacc | jair: i mean, we're trying to see if THP is what is causing your growing memory | 18:27 |
| jair | I will need to let it do it by itself.. :( | 18:27 |
| nacc | jair: so disable it at t he grub config | 18:27 |
| nacc | jair: and reboot | 18:27 |
| teward | ^ this | 18:27 |
| nacc | jair: i mean, i guess you're welcome to wait, but it doesn't tell us anything | 18:28 |
| nacc | in and of itself | 18:28 |
| dpb1 | jair: is this a production server? | 18:28 |
| teward | if it is you're better off rebooting *now* rather than letting it 'die' on its own | 18:28 |
| jair | nacc: understand but this is a prod router providing BGP to our organization I need to wait until reboot itself | 18:28 |
| teward | jair: and your organization can't have a short period of downtime for 'emergency maintenance'? | 18:29 |
| jair | dpb1: yes unfortunately I am trying to help the organization but I am not the main boss | 18:29 |
| teward | if *that* is the case you have a bigger issue than just THP being enabled | 18:29 |
| jair | teward: believe me it's complicated | 18:29 |
| dpb1 | jair: dude, insert testing in production meme here. :) | 18:29 |
| teward | jair: i know what 'complicated' is, i'm an IT consultant for several businesses on my own, as well as employed by others directly, all in the IT role. | 18:29 |
| jair | guys I know and I can't agree more , but I am not the boss unfortunately | 18:30 |
| teward | but when things need emergency-fixed the companies tolerate a short period of downtime :P | 18:30 |
| teward | jair: so talk to the boss. | 18:30 |
| dpb1 | jair: if it were me, I would reboot, always better to be in control of downtime | 18:30 |
| teward | ^ this | 18:30 |
| jair | only someone I don't know what they are thinking put a 17.10 in a prod server | 18:30 |
| dpb1 | jair: yes, that is also a fail | 18:30 |
| jair | yep | 18:31 |
| nacc | jair: wait, so your compnay is ok with random reboots | 18:31 |
| nacc | jair: but not with planned reboots? | 18:31 |
| teward | ^ this is what i was saying | 18:31 |
| teward | if that is the case the company is fubar with policies,. | 18:31 |
| nacc | yeah :) | 18:31 |
| teward | jair: take a page from me: | 18:31 |
| teward | ***talk to your boss*** about an emergency reboot | 18:31 |
| teward | they can probably tolearate 5 minutes of planned downtime vs. two hours as a result of a random crash | 18:32 |
| teward | *just saying* | 18:32 |
| jair | yes doing that now | 18:32 |
| jair | hold on please | 18:32 |
| Ussat | teward, I used that arguement in a meeting with the IT management once.....they were concerned about giving me downtime.....I just sat back and said "No Problem, when it crashes we will fix the issue". I got my downtime | 18:33 |
| nacc | +1 | 18:34 |
| teward | Ussat: and that's usually my argument as well. And most of the outages I cause are no more than 20 minutes of downtime, and that part is usually when I'm just rebooting the SAN for emergency maintenance or have to reboot it to put drive expansion arrays on it | 18:34 |
| teward | so... :p | 18:34 |
| sdeziel | for some reasons, I (wrongly?) assumed that THP was something you could disable live and the kernel would breakup the huge pages into multiple regular ones | 18:36 |
| trippeh | you can disable it but I dont thing that changes existing huge pages | 18:37 |
| trippeh | think | 18:38 |
| nacc | sdeziel: yeah, you can disable it | 18:38 |
| nacc | sdeziel: i think there might be a way to release the pages manually, but i don't know | 18:38 |
| sdeziel | nacc: trippeh: testing it as we speak | 18:39 |
| jair | friends I am really sorry I am in Japan now and it is 3:39 am | 18:39 |
| sdeziel | so far AnonHugePages reduced a little | 18:39 |
| jair | I will let the server crash not say anything and report if that change fixed the issue | 18:39 |
| trippeh | I should test if THP is OK for $job workloads again one of these days. I've usually been left disappointed. | 18:40 |
| nacc | jair: sorry, i might be totally wrong | 18:40 |
| rbasak | nacc: o/ | 18:40 |
| nacc | jair: i was doing more research (it's been a while since i was libhuge maintainer :) | 18:40 |
| nacc | jair: DirectMap1G is a reflection of the TLB status | 18:40 |
| jair | nacc: OK | 18:40 |
| jair | should I enable that back? | 18:40 |
| nacc | jair: can you pastebin /proc/meminfo again? | 18:41 |
| jair | ok | 18:41 |
| nacc | rbasak: have a few minutes for a HO? | 18:41 |
| rbasak | nacc: sorry, about to eat dinner | 18:41 |
| nacc | rbasak: np, i think i found a few gotchas in source_builder | 18:41 |
| nacc | i'm fixing them in my branch, but they'll need your review for sure :) | 18:41 |
| rbasak | nacc: OK | 18:41 |
| nacc | rbasak: enjoy your evening! | 18:41 |
| rbasak | Thanks! | 18:41 |
| nacc | jair: i think that menas your kernel is using 1G pages for something | 18:43 |
| jair | nacc: here https://paste.ubuntu.com/p/MkkpnxySxR/ | 18:44 |
| nacc | jair: yeah, so iiuc, 80% of your system memory is being consumed by the kernel for its mappings | 18:45 |
| jair | sorry I got disconnected | 18:46 |
| nacc | i can imagine a networking table using up a ton of space | 18:46 |
| nacc | if the server is being heavily used | 18:46 |
| jair | nacc: so, should I enable back that setting? | 18:46 |
| nacc | jair: do you see that DirectMap1G value increasing? | 18:46 |
| nacc | jair: yeah, it won't have any effect | 18:46 |
| jair | OK | 18:46 |
| nacc | jair: is that meminfo different than the last one you gave me? | 18:46 |
| jair | no increase | 18:47 |
| jair | I can compare | 18:47 |
| jair | hold on | 18:47 |
| jair | I will do a fdiff | 18:47 |
| jair | diff | 18:47 |
| nacc | jair: i am diffing here | 18:47 |
| nacc | jair: hrm, something ate ~400M of memory from the free | 18:48 |
| nacc | jair: but i'm not seeing any equiv. growth | 18:49 |
| jair | nacc: here http://paste.debian.net/1011539 | 18:49 |
| nacc | jair: yes | 18:50 |
| nacc | jair: your system is fully up to date? | 18:50 |
| jair | Yes | 18:50 |
| jair | nacc: here http://paste.debian.net/1011531 | 18:50 |
| sdeziel | so anonhugepages are not deaggregated | 18:51 |
| nacc | sdeziel: good to know :) | 18:52 |
| nacc | sdeziel: but it was a red herring/misapprehension on my part anyways | 18:52 |
| jair | changed back: | 18:52 |
| jair | http://paste.debian.net/1011531 | 18:52 |
| jair | sorry | 18:52 |
| nacc | jair: i'm trying to think of what might be happening | 18:52 |
| nacc | it *seems* likely that something in kernel is reserving the memory | 18:52 |
| nacc | and not freeing it or so | 18:52 |
| nacc | jair: are new iptables rules being writen constantly? | 18:52 |
| jair | I mean this > # cat /sys/kernel/mm/transparent_hugepage/enabled | 18:53 |
| jair | always [madvise] never | 18:53 |
| jair | nacc: nahh | 18:53 |
| jair | this is just doing routing from our ISP to our infrastructure | 18:53 |
| jair | we are using it as router | 18:53 |
| nacc | jair: can you pastebin `cat /proc/mounts` ? | 18:54 |
| jair | the only reason I got from the guy who installed 17.10 in the R440 dell was because he could not install LTS server because did not supported the raid controller perc 740P | 18:54 |
| jair | he could not see the drives | 18:54 |
| jair | ok | 18:54 |
| jair | nacc: http://paste.debian.net/1011542 | 18:55 |
| nacc | jair: to be clear, MemFree being low is normal | 18:56 |
| nacc | you want all your memory to be in use | 18:56 |
| nacc | but MemAvailable decreasing is a bit odd | 18:56 |
| jair | right | 18:56 |
| nacc | jair: do you have a cpature of full console log when the system crashes? | 18:57 |
| nacc | specifically, the *first* oom report | 18:58 |
| jair | nacc: yes | 18:58 |
| jair | nacc: let me pass it | 18:58 |
| jair | nacc: https://ibb.co/g65wkx | 18:59 |
| jair | there | 18:59 |
| nacc | jair: there should be a bit more before that | 19:00 |
| jair | nacc: this is captured from the idrac IPMI tool | 19:01 |
| jair | nacc: I would have to record yje screen or something | 19:01 |
| jair | nacc: perhaps dmesg? | 19:01 |
| jair | it's going to crash soon | 19:04 |
| jair | http://paste.debian.net/1011546 | 19:04 |
| nacc | jair: dmesg will be gone once you reboot | 19:04 |
| jair | hmm | 19:04 |
| nacc | jair: you want to actually grab the console the whole time | 19:04 |
| jair | I see | 19:05 |
| jair | I wonder if that is possible in idrac ipmi | 19:05 |
| nacc | jair: well, use a typescript and hook into a screen session or so? | 19:05 |
| jair | naac I think we will install hwe kernel and install 16.04 that is what we are all advising | 19:06 |
| jair | I got this | 19:06 |
| nacc | jair: well, i mean the hwe kernel on 16.04 is the same as the kernel on 17.10 right now | 19:08 |
| nacc | afaik | 19:08 |
| jair | https://answers.launchpad.net/ubuntu-certification/+question/664756 | 19:08 |
| nacc | jair: i mean, yes you should use the 16.04 release anyways | 19:09 |
| jair | check the options Jeff Lane gave me | 19:09 |
| nacc | yes i'm reading | 19:09 |
| nacc | jair: when the oom killer runs, it emits a bunch of data about the state of memory | 19:10 |
| nacc | including all pages currently allocated | 19:10 |
| nacc | that is what you need to obtain to debug what is happening | 19:10 |
| nacc | jair: i would guess you'll see the same issues with 16.04.3, but that would be a good thing to test | 19:10 |
| jair | Yep because Dell say in their website that 16.04 is supported | 19:11 |
| jair | well nacc Thank you so much for battling with me on this | 19:12 |
| nacc | jair: yw | 19:13 |
| DammitJim | my syslog is printing this: smbd.service: Got notification message from PID 12210, but reception is disabled. | 19:16 |
| DammitJim | is this something to worry about or is it benign? | 19:16 |
| jair | bight night | 19:17 |
| dpb1 | new vampire flick | 19:20 |
| sarnold | dpb1: hehe :) | 19:22 |
| sarnold | but with "bight", it'd be vampire pirates. it's a rope joke. | 19:22 |
| teward | heh | 19:26 |
| nacc | powersj: ping | 19:45 |
| powersj | nacc: back? | 20:20 |
| nacc | powersj: yeah, sorry, power hiccup | 20:21 |
| nacc | i think i figured out my issue | 20:21 |
| powersj | ok :) | 20:21 |
| nacc | powersj: codecoverage plugin to pytest creates a file | 20:21 |
| powersj | yeah | 20:21 |
| nacc | i want to avoid doing that with the self-test, since we don't know where we're runnig from | 20:21 |
| === chat is now known as Guest64380 | ||
| === keithzg_ is now known as keithzg | ||
| === Epx998- is now known as Epx998 | ||
| nacc | powersj: please rerun the new CI on https://code.launchpad.net/~nacc/usd-importer/+git/usd-importer/+merge/336877 | 22:58 |
| nacc | powersj: err, resubmitted so https://code.launchpad.net/~nacc/usd-importer/+git/usd-importer/+merge/338593 | 22:58 |
| nacc | rbasak: --^ fyi, please review that one | 22:59 |
| powersj | nacc: https://jenkins.ubuntu.com/server/job/git-ubuntu-ci-redux/8/console | 23:00 |
| nacc | powersj: thanks | 23:00 |
| nacc | powersj: that should pass | 23:00 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!