[08:43] <vikram_> Hi All
[08:44] <vikram_> I am looking out for a allinone openstack package for openstack
[08:44] <vikram_> can anyone plz suggest some .iso
[09:28] <Walex2> vikram_: given the enormous number of OpenStack components that is a very vague question
[09:29] <Walex2> vikram_: the "standard" way of doing OpenStack on Ubuntu server is via Juju "charms" though. Unfortunately both Juju and OpenStack tend to be amazingly buggy.
[09:44] <vikram_> Walex2: oh
[09:44] <vikram_> Walex2: I just want a basic openstack deployment model
[09:46] <vikram_> i remember one ubuntu package for icehouse was released earlier
[10:31] <ducasse> I'm setting up a home file server and was thinking of using ZFS to mirror the main data disks, but the machine hasn't got ECC memory. Should I still go with ZFS, or would it be better to use mdadm + ext4 or maybe btrfs?
[11:36] <RickyB98> hello :)
[11:36] <RickyB98> i configured sasl on postfix and dovecot, but my client can still use SMTP without logging in.. authentication works, but no login works too.. how can i restrict that?
[11:39] <RickyB98> also, i configured mail to use Maildir/, doing MAIL=/home/myuser/Maildir/, but when i open an email with mail from mailutils, they get deleted and saved to mbox.. what's the variable to change that?
[11:50] <RickyB98> anyone here?
[11:54] <rbasak> !patience | RickyB98
[11:55] <RickyB98> i'm not repeating your question..
[11:55] <RickyB98> my question*
[11:55] <rbasak> "if nobody knows your answer, nobody will answer you"
[11:56] <rbasak> Sorry I can't tune the bot's reply for the exact situation. It's close enough and I hope still helpful.
[11:56] <RickyB98> however, someone in #ubuntu told me to come here.. now i was being answered there, should i go back there?
[11:56] <RickyB98> going to idle here anyway, not gonna run away after 5 minutes :P
[12:12] <ziyourenxiang> RickyB98: by client do you mean an external machine connecting to yoru postfix over SMTP and relaying?
[12:13] <RickyB98> client as in a mail client, could be thunderbird or anything
[12:13] <RickyB98> i'm using mac's default atm
[12:22] <matt_dupre> ducasse: I've been running ZFS without ECC for years and never noticed any problems.  I'm not really qualified to judge the risk vs non-ECCed mdadm / btrfs, but personally I wouldn't consider it a factor in deciding between them.
[12:23] <ducasse> matt_dupre: OK, thanks. Is it a good choice for a small, home fileserver?
[12:37] <matt_dupre> ducasse: I like it: I actually run on FreeBSD, but I hear ZFS on Linux has come on a long way.  I think the biggest weakness is that it can be tricky to add disks later on.
[12:38] <ducasse> matt_dupre: I though one of it's strengths was that is was easy to expand by adding disks, but that might not be true for mirrors - I don't know.
[12:38] <matt_dupre> (For example, you can't add a disk to a RAIDZ, so you can't just buy an extra disk every couple of years.  Big upgrades (e.g. doubling capacity) are usually simpler.
[12:38] <matt_dupre> Yeah, mirrors are easier
[12:39] <ducasse> matt_dupre: that's what I will be doing. If I set up 2x3TB now I can expand with two new disks later, right?
[12:39] <matt_dupre> Yeah, you can go from a RAID1 to a RAID10
[12:41] <ducasse> matt_dupre: OK, thanks. I think I'll go with ZFS because there's a possibility I will move to either FreeBSD or FreeNAS later, and then I should be able to just export/import the pool, according to everything I've read... Besides, I'm not sure if btrfs is really a good alternative yet.
[12:42] <matt_dupre> Yeah, as long as you take care to create the pool at the minimum version supported across everything.
[12:44] <matt_dupre> Other bit of creation-time advice is to try to get the right ashift (google it).  Your drives are probably 4k sectors internally, but report as being 512byte.
[12:51] <ducasse> matt_dupre: I know about the ashift thing, all drives use 4K physical, 512 logical, so I'll use 12. Also I think FreeBSD is generally ahead of ZoL in what feature flags are supported, so that should be fine. Thanks a lot for your advice!
[13:14] <coreycb> jamespage, I've been struggling for too long to get hypothesis test failures situated, so I'm going to drop hypothesis from python-cryptography for now.  It only affects one test function.
[13:14] <jamespage> coreycb, okay
[13:32] <wojtczak> my ubuntu server doesnt boot, I've installed it via debootstrap on lvm and raid. made sure to configure network, grub in chroot. now the server doesnt respond, back in rescue mode I don't see network interfaces in dmesg. I've tried adding ixgbe module to /etc/modules but this does not help. https://gist.github.com/sabcio/5a8dea26ecee26af5fa5
[13:46] <MACscr> ok, so im booting servers through pxe iscsi root and using the same initdrd.ig for all servers. The problem is that i need a unique /etc/iscsi/initiatorname.iscsi in initrd for each host. Any way to dynamically create it? ive seen ways to do it with sysconfig on a redhat based system, but no clue for ubuntu/deb
[13:48] <Walex2> wojtczak: you may have to add them to the 'initrd'
[13:48] <MACscr> Walex2: was that meant for me or was someone else talking about initrd before i got in here?
[13:49] <wojtczak> Walex2: thx, something to google about
[13:49] <MACscr> ah, guess you were. lol
[14:20] <coreycb> jamespage, can you sponsor my changes for xenial?  https://git.launchpad.net/~corey.bryant/ubuntu/+source/python-cryptography/
[14:21] <coreycb> jamespage, it builds successfully
[15:00] <smoser> magicalChicken, rharper i tried to write some of the things i think are next on the list for curtin at
[15:00] <smoser>  https://public.etherpad-mozilla.org/p/curtin-work-201512
[15:01] <magicalChicken> smoser: Nice
[15:01] <magicalChicken> smoser: I had started doing some work on separating logic between block and block meta last night
[15:02] <magicalChicken> smoser: https://code.launchpad.net/~wesley-wiedenmeier/curtin/partitioning-cleanup
[15:04] <smoser> looking
[15:09] <coreycb> zul, can you sponsor this for xenial?  https://git.launchpad.net/~corey.bryant/ubuntu/+source/python-cryptography/
[15:10] <zul> coreycb: yeah lemme finish what im doing first
[15:10] <coreycb> zul, sponsoring it
[15:10] <coreycb> :)
[15:11] <coreycb> zul, I dropped hypothesis from python-cryptography for now because it's tests don't run and I've had trouble getting them running successfully.  It only affects one test function for python-cryptography.
[15:11] <smoser> magicalChicken, why 'extra_init' ?
[15:12] <magicalChicken> smoser: Ah, yeah, I know that's a little ugly
[15:12] <magicalChicken> smoser: The idea was that the BlockDevice class could be the parent of Disk as well as Partition
[15:12] <smoser> well, i agree with that.
[15:12] <magicalChicken> smoser: And possibly the virtual filesystem layers as well, and I wanted to keep as much common functionality in the super class as possible
[15:13] <smoser> but super() is the typical way of doing that. no?
[15:13] <magicalChicken> smoser: Yeah, I guess that would probably be better, it's probably not great to try to just have one __init__
[15:15] <smoser> and then generally, when you're doing something.. unless you have a reason to merge with a branch from somewhere other than trunk, you should avoid that.
[15:15] <smoser> to just keep a branch doing 1 thing.
[15:15] <magicalChicken> smoser: Yeah that makes sense
[15:16] <magicalChicken> smoser: I was thinking I could add a Disk.wipe() function that could call the block.wipe_volume function
[15:16] <magicalChicken> smoser: But yeah, it would probably have been better to wait to merge that
[15:16] <smoser> right.
[15:17] <smoser> the other htig is that we have 2 types of block devices. one that is "to be done" and one that is existing now.
[15:17] <smoser> BlockDevice(dev="/dev/vda")
[15:18] <magicalChicken> smoser: yeah, that makes sense
[15:18] <smoser> BlockDevice(config={'id': 'abc', 'type': xx}
[15:18] <magicalChicken> smoser: This code should handle that okay though I think
[15:18] <magicalChicken> smoser: You can create a disk based on the path it already has
[15:19] <magicalChicken> smoser: maybe a function to determine the difference between a config and what exists would be good
[15:19] <zul> coreycb: done
[15:21] <smoser> magicalChicken, right. essentially a factory
[15:22] <rharper> smoser: reading
[15:23] <magicalChicken> smoser: I was also thinking instead of going through config elements one at a time we could unflatten the config into a hirearchy of BlockDevice instances
[15:23] <magicalChicken> smoser: And apply the config from the top down
[15:23] <smoser> i think that is generally necessary, yes
[15:24] <magicalChicken> smoser: That should be faster, because we could create all the partitions in one go and keep track of state on everything
[15:24] <rharper> if we have the order, it'
[15:24] <rharper> it's also possible to run some of those in parallel
[15:25] <rharper> no reason I can create a partition on 5 different devices at the same time either (though that's performance enhancement for later)
[15:25] <smoser> i didnt realize we were creating each partition individually.
[15:25] <smoser> stil
[15:25] <magicalChicken> rharper: yeah, that would be pretty nice
[15:25] <smoser> yeah, ./split-disk.py /dev/vdb 32
[15:25] <smoser> is taking a while :)
[15:25] <rharper> hehe
[15:25] <coreycb> zul, thanks, appreciate it
[15:25] <magicalChicken> smoser: Yeah, the way the partitions are made right now is kinda slow...
[15:26] <smoser> rharper, i'm not opposed to parallel. but i'm not happy with the predictability of the whole system as it is . i dont feel a strong need to add more to races to it. :)
[15:26] <rharper> smoser: magicalChicken, also, I think we need to look again at the udevadm settle bits, there are some extra flags like the --exit-if-exits=/path/to/file/
[15:27] <rharper> smoser: for sure; I'm highlighting some opportunity once we switch away from in-order handling
[15:27] <magicalChicken> rharper: yeah, that would definitely help the current devsync fix
[15:28] <magicalChicken> rharper: also, refactoring like this would mean that we could keep track of whether or not we've synced with udev for each device
[15:34] <smoser> hm.
[15:38] <nat0> Is there any debian-installer instruction that can be passed thru a preseed file to tell anna-installer not to check for upgrade paths on the repo mirror?
[15:43] <nat0> It's truly bizarre how this is only a problem when deploying new ubuntu machines and not vanilla debian ones...
[15:45] <smoser> nat0, sorry, cant help.
[15:45] <nat0> ;_;
[16:23] <smoser> rharper, magicalChicken so, i never recognized this before
[16:24] <smoser> but there is a device node availability limitation that means more than 16 partitions is odd.
[16:24] <smoser> ie, /dev/vda16 wont get created nor will /sys/class/block/vda16 exist
[16:24] <rharper> yeah, the minor number space right ?
[16:24] <smoser> right
[16:24] <rharper> yeah, hence lvm
[16:24] <magicalChicken> wow, I didn't know that
[16:25] <magicalChicken> so should we check if the partitioning config has more than 16 parts on a disk?
[16:25] <magicalChicken> or is that something that we should trust the config generator to do?
[16:26] <patdk-wk> would be more than lvm, should apply to all block devices
[16:26] <patdk-wk> such as putting >16 partitions into a gpt disk
[16:28] <smoser> http://paste.ubuntu.com/14029368/ <-- some info on it.
[16:28] <smoser> patdk-wk, right. thats just what i did ^. i put 20 partitions on a gpt disk, and only 15 of the partitions have block devices.
[16:29] <rharper> patdk-wk: lvm and devicemapper can work around it by allowing new device node space to map to other parts of the disk
[16:29] <patdk-wk> oh, he meant lvm as a solution :)
[16:29] <rharper> that's why I mentioned lvm
[16:29]  * patdk-wk doesn't like lvm
[16:29] <RoyK> patdk-wk: why not?
[16:30] <patdk-wk> it goes ontop of the devicemapper stack
[16:30] <patdk-wk> and I have noticed nothing but write latency issues when I use that
[16:30] <RoyK> so what if it works?
[16:30] <patdk-wk> it seems to buffer my writes for a few seconds
[16:30] <patdk-wk> causing horrible issues on some of my servers, and on my workstation
[16:30] <patdk-wk> remove lvm/devicemapper, no more random stalls while it writes
[16:30] <RoyK> patdk-wk: then something is probably bad somewhere else - I'm using it on 150 servers and it works well for me ;)
[16:31] <patdk-wk> heh? you should know that is not the definition of *it works*
[16:31] <patdk-wk> the ONLY change, was to remove lvm
[16:31] <patdk-wk> and the problems went away
[16:31] <patdk-wk> no hardware changes
[16:31] <patdk-wk> nothing else
[16:32] <smoser> interesting, even if you worked around this limitation by having dm magically create device nodes named /dev/vdb16 that did what it *should* do it wouldnt appear the same in /sys/class/block as /dev/vdb15 does . the reader would have to know about dm to understand it.
[16:32] <patdk-wk> had the issue here on ubuntu 10.04/12.04, and on rhel5
[16:32] <RoyK> patdk-wk: I've never seen 2-3secs latency
[16:32] <patdk-wk> royk, there are kernel settings for it
[16:32] <RoyK> patdk-wk: what sorts?
[16:32] <patdk-wk> they might have changed, I had adjust them, but never could find ones that seemed to work better
[16:33] <RoyK> patdk-wk: got any docs on this?
[16:33] <patdk-wk> looking
[16:33] <patdk-wk> back when I cared to fix this issue, I did :)
[16:33] <patdk-wk> but been a few years
[16:33] <RoyK> patdk-wk: it just seems strange, after all, centos/rhel has been shipping with lvm by default for years
[16:34] <patdk-wk> yes, and I was having all kinds of issues on a very busy webserver
[16:34] <patdk-wk> removed that lvm from it, issues went away
[16:35] <RoyK> strng
[16:36] <patdk-wk> but it seemed to me, if I remember write, it was getting buffered twice into the write buffer or something like that
[16:36] <patdk-wk> adjust vm.dirty_writeback_centisecs would help to a point
[16:38] <patdk-wk> heh, cannot remember the right things
[16:38] <patdk-wk> but I had the isuse on several different machines, the result was always the same, it would feel like the machine just froze up for several seconds, while it dumped a bunch of writes to disk, then go back to normal
[16:39] <patdk-wk> and only happened with using devicemapper
[16:50] <smoser>  https://bugs.launchpad.net/curtin/+bug/1526437 <-- magicalChicken rharper
[16:51] <magicalChicken> smoser: Makes sense
[16:51] <magicalChicken> smoser: I can add a check for that in partition_handler sometime today
[16:55] <smoser> magicalChicken, give yourself a name at $ sudo /tmp/even-partition /dev/vdb 20
[16:55] <smoser> size=40960M numparts=20 partitions_of=2047M label=gpt dev=/dev/vdb
[16:55] <smoser> curstart=2048 curend=4194304 part=0
[16:55] <smoser> curstart=4194304 curend=8386560 part=1
[16:55] <smoser> curstart=8386560 curend=12578816 part=2
[16:55] <smoser> curstart=12578816 curend=16771072 part=3
[16:55] <smoser> curstart=16771072 curend=20963328 part=4
[16:55] <smoser> curstart=20963328 curend=25155584 part=5
[16:55] <smoser> curstart=25155584 curend=29347840 part=6
[16:55] <smoser> curstart=29347840 curend=33540096 part=7
[16:55] <smoser> curstart=33540096 curend=37732352 part=8
[16:55] <smoser> curstart=37732352 curend=41924608 part=9
[16:55] <smoser> curstart=41924608 curend=46116864 part=10
[16:55] <smoser> curstart=46116864 curend=50309120 part=11
[16:55] <smoser> curstart=50309120 curend=54501376 part=12
[16:55] <smoser> curstart=54501376 curend=58693632 part=13
[16:55] <smoser> curstart=58693632 curend=62885888 part=14
[16:55] <smoser> curstart=62885888 curend=67078144 part=15
[16:55] <smoser> curstart=67078144 curend=71270400 part=16
[16:55] <smoser> curstart=71270400 curend=75462656 part=17
[16:55] <smoser> curstart=75462656 curend=79654912 part=18
[16:55] <smoser> curstart=79654912 curend=83884032 part=19
[16:56] <smoser> created 20 partitions on /dev/vdb
[16:56] <smoser> $ ls -l /sys/class/block/vdb* -d
[16:56] <smoser> lrwxrwxrwx 1 root root 0 Dec 10 18:58 /sys/class/block/vdb -> ../../devices/pci0000:00/0000:00:04.0/virtio2/block/vdb
[16:56] <smoser> lrwxrwxrwx 1 root root 0 Dec 15 16:25 /sys/class/block/vdb1 -> ../../devices/pci0000:00/0000:00:04.0/virtio2/block/vdb/vdb1
[16:56] <rharper> heh
[16:56] <smoser> lrwxrwxrwx 1 root root 0 Dec 15 16:25 /sys/class/block/vdb10 -> ../../devices/pci0000:00/0000:00:04.0/virtio2/block/vdb/vdb10
[16:56] <smoser> lrwxrwxrwx 1 root root 0 Dec 15 16:25 /sys/class/block/vdb11 -> ../../devices/pci0000:00/0000:00:04.0/virtio2/block/vdb/vdb11
[16:56] <smoser> lrwxrwxrwx 1 root root 0 Dec 15 16:25 /sys/class/block/vdb12 -> ../../devices/pci0000:00/0000:00:04.0/virtio2/block/vdb/vdb12
[16:56] <smoser> lrwxrwxrwx 1 root root 0 Dec 15 16:25 /sys/class/block/vdb13 -> ../../devices/pci0000:00/0000:00:04.0/virtio2/block/vdb/vdb13
[16:56] <smoser> lrwxrwxrwx 1 root root 0 Dec 15 16:25 /sys/class/block/vdb14 -> ../../devices/pci0000:00/0000:00:04.0/virtio2/block/vdb/vdb14
[16:56] <smoser> lrwxrwxrwx 1 root root 0 Dec 15 16:25 /sys/class/block/vdb15 -> ../../devices/pci0000:00/0000:00:04.0/virtio2/block/vdb/vdb15
[16:56] <smoser> lrwxrwxrwx 1 r
[16:56] <rharper> magicalChicken: I'd wait until we're building the config hierarchy, we'll instantly know then (rather than just checking if the current partition exceeds the limit)
[16:56] <smoser> oh crap
[16:56] <smoser> shame on smoser
[16:56] <smoser> shame
[16:56] <smoser> https://public.etherpad-mozilla.org/p/curtin-work-201512
[16:56] <magicalChicken> rharper: Yeah, that makes sense
[16:56] <magicalChicken> smoser: lol
[16:58] <smoser> magicalChicken, so i think the first thing to do is to get "create a set of tests that run multiple storage configs serially"
[16:58] <magicalChicken> smoser: Yeah, that makes sense
[16:58] <smoser> with minimal cleanup or improvement involved
[16:59] <magicalChicken> smoser: Maybe the best thing to do would be to just write a script to do multiple installs in a vm and communicate with it via http or something
[16:59] <smoser> and then we will heavily rely on those tests to ensure that otherthings are working well.
[16:59] <smoser> i think we specifically want to shortcut 'install'
[17:00] <rharper> smoser: we also talked about running multiple curtin install commands
[17:00] <magicalChicken> yeah, 90% of the time is spent doing extract/curthooks right now
[17:00] <rharper> it could be a payload of configs, and running curtin install on each one in sequence
[17:02] <smoser> rharper, testing suport for how subiquity calls curtin is important. i'll agree with that.
[17:02] <magicalChicken> I think that one platform could be used for both goals though
[17:02] <smoser> i just want to test storage config specifically.
[17:02] <magicalChicken> we just need a way to start a vm, then from a test script pass in configs to partition or configs to install
[17:04] <smoser> i'm not opposed at all to running 'nosetests tests/vm-storage-tests/' inside a vm
[17:04] <magicalChicken> yeah, that would work pretty well
[17:04] <magicalChicken> and we could just have a data partition like with vmtests where it could write it's results as it runs
[17:05] <teward> anyone willing to sanity-check the nginx merge debdiff for me, out of curiosity?  Hate to ask, but as I've been working on this since 11:00 yesterday (with an 8 hour break overnight for sleep!)... it could use another check
[17:05] <smoser> right.
[17:06] <magicalChicken> I can get started on that now, I can't really find any decent way to identify a disk to udev on Trusty
[17:07] <smoser> teward, if you pastebinit, i can say "that looks too long for me to review right now".  ie, i can take a very cursory look.
[17:07] <teward> smoser: would an uploaded-to-launchpad link work?  https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1510096/+attachment/4535172/+files/nginx-merge_debian-vs-ubuntu_nginx_1.9.6-2_1.9.6-2ubuntu1.debdiff
[17:07] <teward> SILENCE, bug bot >.<
[17:08] <smoser> magicalChicken, lets try to separate out the copying of results somewhere from the running of the tests.
[17:08] <smoser> and even separate the tests by number of disks required or something.
[17:08] <teward> smoser: it *should* be fine, the larger merge debdiff from 1.9.3 -> 1.9.6 aincluded is on the bug
[17:08] <magicalChicken> sure, okay
[17:08] <teward> (but that debdiff there compares Debian to the merge itself, to see what's actually changed there)
[17:09] <smoser> as ideally i'd like to be able to launch a vm somewhere, and then just type: nosetests tests/vm-storage-config
[17:09] <smoser> and skipIf@ the ones that aren't going to run for me.
[17:09] <smoser> teward, i was afraid you were going ot say 'merge' :)
[17:09] <teward> smoser: it is a merge :)
[17:10] <smoser> always so hard to do that.
[17:10] <teward> smoser: i've got both debdiffs present, force of habit
[17:10] <teward> it builds.  it runs.  it IDs as the updated software.
[17:10] <teward> and because Sec team mandates it, HTTP/2 is disabled
[17:10] <teward> the headache is it expands the delta
[17:10] <smoser> have you looked at all at rbasak's process for merges ? its really good. separating out the 'logical ubuntu delta' into a set of patches.
[17:10] <smoser> https://github.com/basak/ubuntu-git-tools
[17:11] <teward> yeah i have, except I've never been able to get a handle on his process...
[17:11] <teward> and the 'logical ubuntu delta' is the one i did link here
[17:11] <teward> (all the stuff from the previous merges included, plus additional *new* mandates from the sec team)
[17:12] <teward> (i.e. the Ubuntu branding, ubuntu-core added, the apport hooks we had to add for Wily+...
[17:12] <teward> i should probably put this down for a day and relax :/
[17:12] <teward> i can wait for rbasak to once-over it
[17:12] <teward> though i can easily upload direct :P
[17:16] <smoser> teward, you seem to have done a careful job, but i think i dont have time to review at the moment.
[17:23] <smoser> magicalChicken, do you have a reason taht you have used 'partprobe' rather than 'blockdev --rereadpt' ?
[17:25] <magicalChicken> smoser: not really, I assumed they did the same thing pretty much
[17:25] <magicalChicken> smoser: I know partprobe triggers the udev events for creating the block device
[17:29] <smoser> yeah. they are definitely different. and partprobe is smarter i think.  watch 'udevadm monitor' and run each. partprobe generates a lot more interaction.
[17:29] <smoser> i think the difference is that partprobe is actually reading the partition table itself, and telling the kernel about it. where rrpart is just telling the kernel "hey, reread the table there".
[17:30] <magicalChicken> smoser: that makes sense
[17:30] <magicalChicken> smoser: I guess we don't actually need to read the table ourselves then
[17:31] <magicalChicken> smoser: I think once we stop doing everything in steps and write the whole partition table in one go then we won't really need to do that anymore
[17:33] <smoser> magicalChicken, well just suspect we dont need to do it now.
[17:34] <magicalChicken> tjat
[17:34] <magicalChicken> *that's probably true
[17:34] <smoser> tjat
[17:34] <smoser> well said
[17:34] <magicalChicken> lol
[19:42] <smoser> magicalChicken, one other thing 'll add to thath pad.
[19:43] <smoser> nicder subprocess execution output log
[19:43] <smoser> http://paste.ubuntu.com/14025172/
[19:44] <magicalChicken> smoser, yeah that would be nice to do
[19:48] <magicalChicken> smoser, we don't need to show all stdout when curtin fails, we could always write stdout to a tmp file, but including it in the logs takes up a lot of space
[19:49] <smoser> i think we want it in the log. i dont care about space. i just want to easily see the error.
[19:50] <magicalChicken> okay, yeah. it shouldn't take too much work to get it to print with proper formatting