Still another Opensource/Crypto blog

Thanks again Markos for the nomination!

I know I’m still somewhat new to be running for Council, but I do hope you will give me at least a little consideration. I have been active within the project for a year as of this month, and an active dev since October. That gives me a somewhat unique blend of knowing the culture of the project, knowing it’s problems, and still being delusional enough that I want to try to fix some of them. Since joining I have run into a couple things that I would like to see resolved sooner rather than later.

I have had a strong interest in Crypto for a long time (over 10 years at this point, and I’m only 23!), and one thing that I’ve noticed Gentoo is lacking is any form of security whatsoever between us, and our user base. Given that, I’d really like to see (at the very bare minimum) GLEP 58 become a reality within the next couple months. I’d also like to see this extended from that point to full tree signing (however, given the state of things, we stand to gain the most security by implementing GLEP 58, so that should really get done first). I personally plan on tryingto carry this forward (regardless of the Council elections). There will be more on this topic posted to -dev when I get everything I want to say coherent enough that I won’t sound like a babbling idiot.

The next thing I would like to see addressed is some of the current “team” problems. It is imperative for this project to have good working QA and DevRel teams. We are honestly zilch of a project if we are constantly breaking things / failing to even remotely work together. However, as is very easily seen by the current situation that apparently it is going to take “Council Appointments” for lead positions before these teams will be able to operate correctly without people forever questioning their right to do so. There were a number of proposals along this line at [1].

Lastly, git. It needs to happen. It’s no magic bullet, by I really would love to have the whole repository on my own ruddy machine. Need I say more?

So, for those that want to tl;dr; this:
Summary:
1) GLEP 58-61.
2) Consider QA/DevRel “Team Lead Appointments” or something similar.
3) Git.

Thanks for your time! Feel free to question anything!

[1]

http://archives.gentoo.org/gentoo-dev/msg_e1fd66dc60b2e361950f0c3521eb84ab.xml

So, here’s what I’ve done so far:
1) I adjusted the depend to make sure util-linux[loop-aes] is installed. This should prevent any of this from happening again. It’s not the prettiest solution, but it is the most functional that I’ve been able to come up with.
2) The patches for 2.19.1 are out, the bug is open here. Hopefully that will get added in soonish. It will be a little bit till it’s keyworded, but at least it will be in the tree.

Notes for those affected:
If you’re using a util-linux from an overlay, you’ll be fine. You don’t need to anymore though unless you require something from util-linux-2.19.1. If you want to switch back to the version in portage, just remove the overlay and run emerge -u world. It *will* downgrade your util-linux to the last one with loop-aes support (2.18-r1). This is intentional.

If your box is still broken: just emerge -u world. It will fix it by downgrading.

I’m going to look into another solution to make us less dependent on util-linux (at least to the point that a util-linux without loop-aes support won’t make your data inaccessible). That may be a little while in the coming though.

As to it being Treecleaned, we definitely have enough users of this to keep it around. I’m not personally convinced that it is the way to go at all. I would love to see some form of data to back up the “it’s more secure” logic, mostly because I’m curious, and partly because I don’t like crypto claims like that without a bit of evidence. Having said that just because there is something newer/shinier isn’t reason to remove this. I will say though, that I think this one will join the ranks of truecrypt in terms of my headaches. Dm-crypt is nice since there is little magic going on outside of making sure you have device-mapper in the kernel, and that cryptsetup is installed. No patches, no magic foo, and virtually no headache for me. Until I see some serious data suggesting there is an issue with dm-crypt, I will keep encouraging people to look into dm-crypt and consider switching. If there is something that really makes loop-aes > dm-crypt though, I would love to know about it =).

I’m sorry for the headache around all of this. If all goes well, we should be in the clear from here on out. This shouldn’t repeat itself.

A while back, I wrote something about Truecrypt, mostly trying to get a feel for the number of people still using it, and why they were. And now it’s time to ask some similar questions about loop-aes. Currently there are two bugs about loop-aes, bug 370635 and bug 354451, that are bothering me.

Personally, I’m a little torn on this particular issue. I am not a huge fan of having lots and lots of different disk encryption utilities in the tree. First, when they break, people get very very upset, and understandably so. If your data is in one of those encrypted disks and you can’t get to it because of bug x, you have a right to be snarky. But the more of them that there are, the more work it is to keep all of them in pristine working order.

Loop-AES to me seems to have a few disadvantages. The biggest I’m seeing now is that it requires patching to util-linux, which requires the patch to be working and playing nice. Util-linux is a highly important packages, and waiting to bump a version because loop-aes isn’t ready, really isn’t much of an option. This is what led to both of the bugs at hand. Now, of course I can put blockers in, and that is decidedly an option, but first, I want to know why people are choosing to use loop-aes over dm-crypt.

Is this something we need/want to keep in the tree? Or is it merely causing more headaches than it’s worth. Candidly, I’d love to punt TrueCrypt. It’s not a lot of fun to deal with. The build system isn’t particularly pleasant, and upstream isn’t particularly.. helpful. But, I heard a number of decent reasons why we should try to keep it around. Now I want to know why I should spend time maintaining loop-aes?

If there really aren’t any good reasons, I’m thinking it might be time to start urging people to migrate to dm-crypt. Soon, cryptsetup will support loop-aes disks, so getting at the data won’t be a problem, but it won’t be near as fast / native. The simple fact is that, as far as I’ve been able to tell, dm-crypt is decidedly better supported, a lot more flexible, and a lot easier to deal with. Device-mapper is native to the kernel, no patching necessary. You have access to every cipher the kernel does, and the utility to mount dm-crypt devices is a separate application. All of these make both users lives (and my life) easier.

So here it is. Are there reasons to keep this around? Or is it just another crypto app that is being kept in the tree merely for historic reasons?

Well, I’m finally part way through my transition to my new VPS. Basically it boils down to “Same old look, better everything else.”

Formerly, I was using a Link2 from vpslink. I had 1 core, 128MB of RAM, and I paid around $14 a month for it. A few months ago, a friend found prgmr. All I can say is I’m highly satisfied for a lot of reasons. First and foremost, I have direct Xen access. What that meant for me was the ability to boot into the rescue image, and yep, you guessed it, install Gentoo. So now, instead of using Debian for my webserver (which I don’t know as well), I’m using Hardened Gentoo with SELinux. Big step up in security and familiarity. Can’t beat that. The next big perk, I have 2 cores, and 1G of RAM. And, it doesn’t cost near what you’d think. To get something like that from vpslink would have been around $70. From prgmr, it’s $20. Translation: I can finally compile things on my webserver without being able to take a nap in between each package.

Some changes people should be aware of:
I have completely removed the registration ability. When I was transferring wordpress over from the old server, I took a look through the database. Lots and lots of spam users. Given that you don’t need to be a user to post, I don’t see a reason to keep that functionality around. So it’s gone =P.

I’ve also effectively disabled the ability to log in. The wp-admin section of my site is now only available over https, and, well, Apache is now authenticating the client, aka me. This also meant I was able to better restrict access to things I don’t want people poking at, like phpmyadmin.

In the end, I think this will be a lot better. I’ll be tweaking and making adjustments for a while. Feel free to make any suggestions.

C1pher’s Adopt a Package Program (henceforth known as CAPP) aims to help reduce the number of maintainer-needed packages in the Portage Tree. As you can see in this (rather lengthy) thread on gentoo-dev ML, we have a lot of maintainer-needed (from now on, m-n) packages that could really use some adoptive devs.

M-N packages are hard to deal with for a lot of reasons. Ideally we could very easily just punt anything that was m-n from the tree and not worry about it, however, as you can see from that thread, that is anything but the case. Just because something is m-n does *not* mean that it’s broken, or useless, or anything else. What it *does* mean is that when issues show up, it could sit broken for months without anyone looking at it. That is a QA nightmare as well. We don’t like broken packages. We want the tree to be nice and working. However, if that m-n is useful to a fair number of users, or is depended on by other packages, it can lead to headaches… and angered users… leading in more headaches.

Soooooo, hence this post. As much as I would love to adopt all 675 or so packages that are looking for a home, I do *not* have *that* much free time. I’m a geek sure, but I do have a life. So, instead of watching them flounder, I’m offering myself up as a Proxy-Maintainer to anyone would like to adopt a package or 10 from the list. I am very easy to reach via e-mail or IRC, so poke me there and we will work something out.

If you’re curious what packages need some lovin’, here you are. Click away! Cantidates for adoption.

So, as some of you may be aware, I recently went through the process of picking out and acquiring a new laptop. To all of you who helped me narrow my search down to a reasonable number of things, thank you! It helped me out a lot. I know several of you were curious as to how things turned out. Well, here you go!

The Laptop, revisited: It’s a Lenovo ThinkPad X201. I customized mine, since I’m picky as all hell. Here is what I went with:
Processor: Intel Core i5 560M
RAM: 8GB DDR3 PC3-5800
HDD: 500GB 7200rpm
Wireless: ThinkPad b/g/n — More on this to follow
Along with other various tweeks.

So, after a bit of a hassle with UPS, my laptop showed up on Tuesday. I immediately ripped it out of the box, and then proceeded to take it completely apart =P. (What can I say, I’m a geek) which leads me to my first real impression of the laptop. It’s rock solid. And better yet, it’s easy to take apart. That’s a nice big plus for me. If I need to open a laptop up, I’d prefer not have to argue with it for 30 minutes to do so. Lenovo provides the maintenance manuals online and they walk you through everything. As of now, I was able to take it apart last night for a quick change and put it back together in under 4 minutes. Thant’s pretty damn nice for a laptop.

As some of you read before, the ThinkPad b/g/n wireless is not atheros based. It’s in fact a realtek card. So I had purchased an older ThinkPad a/b/g/n card on ebay which was. Well, as it turns out, this wouldn’t actually fit into the space in the case. Not because it was the wrong type of card, but because they have screw down points for a half length mini-pcie card and screw down points for a full length mini-pcie card. The old ThinkPad wireless cards are full length, and they have a bulge on the top and the bottom. The one on the bottom runs into the half length screw down points. So I hopped back onto ebay to order a half length atheros card =). $20 later, I was done.

I then put everything back together, and made sure it still booted. No issues. I reboot, stick in my Gentoo LiveCD (One of the 4938321 that are laying around my apartment. I lose them a lot =P). And start the Gentooification. I’m fairly picky when it comes to my laptop setups. I had drive encryption to deal with, which involved writing Pseudo-Random data to the disk first. It was a 30G root partition, and a 300G home partition. I only did root to begin with. That took 2 hours to begin with. I then proceeded with my install. The reason I went so high on the RAM was so that I could put a 3Gish ramdisk on /var/tmp/portage. That adds a very nice boost to compiling speeds. I got everything updated, and then optimized the ever living crap out of it. I use distcc for a lot, so I can’t use -march=native. Here are my CFLAGS.

CFLAGS=”-O2 -march=core2 -mtune=core2 -mcx16 -msahf -maes -mpclmul -mpopcnt -msse4.2 –param l1-cache-size=32 –param l1-cache-line-size=64 –param l2-cache-size=256 -fomit-frame-pointer -pipe”

I then did a full toolchain recompile.. twice. And recompiled @system a few times after that. Satisfied that things were fairly optimized, I moved on to the rest of the install. Kernel, grub, and several other packages later, I had a booting system. Reboot it, and huzzah. It works. Install X, KDE, and a zillion other packages, and I’m more or less done. The current realtek card worked quite nicely all things considered. However, I live having AP mode and monitor mode, so it will have to go.

Flash forward to last night. My new half mini-pcie atheros card showed up. Reboot to Windows (Sad times, I know, but I don’t like doing BIOS work in *nix) I download the updates to the current BIOS from Lenovo and install them. Now, the X201 has a wlan card white list. I found a plethora of information for getting around that. (Google is your friend =P) As it turns out, while I was working on hacking the white list out myself, I found a .zip on a forum with the already de-white-listed BIOS. So I had been hanging onto that to try. Why waste my time when someone else has done it already =). I flash that modified BIOS, reboot, and pray. It comes back on, no issues. Shut it down again, rip it apart, put the new card in, reboot, and WOOT, it comes back on again. No issues, no white list failures. My adventure was complete. I look in Gentoo, and sure enough, it sees the card, and ath9k is already loaded. (I do custom kernels, so I had already build the module in anticipation.)

And that, is my story. As to the laptop itself, I couldn’t rave about it more. It’s powerful, it’s the perfect size at 12.1″. It weighs a scant 3lb ish. And it has a 9 cell 11 hour battery. I couldn’t ask for more, save maybe a SSD. I’ll get one of those eventually too. Just waiting for some of the newer ones and to try one out in a desktop for a while. Otherwise, perfection. I don’t have a single thing I couldn’t get working. Suspend to RAM is flawless. (I don’t have Suspend to Disk set up yet since I don’t generally use it. I will eventually) I can’t rave about it enough.

If you’re in the market, the X201 is a great way to go for Gentoo if you’re comfortable cracking the case and putting in an atheros card (or going with Intel card to begin with if that does what you want).

Thanks again to everyone that helped out with the research / decision!

Side note: if you decide to get one, and want the BIOS files I use, shoot me an e-mail. I’ll send them to you signed.

First off, thank you to everyone who responded to my last bit. It was greatly appreciated and radically reduced my search sphere.

Given the help, I figured it would be fun to share what I decided.

I ended up going for a custom Thinkpad X201.
Core i5-560M
8GB DDR3 RAM
500GB 7200rm SATA HDD
Thinkpad wireless — More on this to come.
9 cell battery

(Other magic)

If all goes well, it will be to me sometime in the next couple weeks. I do have some interesting foo to share however. The current Thinkpad b/g/n wireless is not Atheros. The older one, Thinkpad a/b/g/n was. The new one is a Realtek card. However, given the price I got the machine for ($1200), I found it hard to change my decision based entirely on that. So I ordered one of the older Thinkpad a/b/g/n cards and am going to try installing that. I know its BIOS uses a wireless card whitelist, however, it leaves one question open.

Will the older Thinkpad a/b/g/n card be on the list? I don’t think it’s too much of a leap to suspect so. It was on all the older X models. So it is a reasonable assumption.

If it isn’t, I know of a couple tricks I can try. 1) Spoof the Vendor/item ID on the card I bought to be one that is on the list and then just trick Linux into using the right module. 2) Hack the BIOS in the true way. An option I have already explored, and one that I think is quite viable. I’m hoping I don’t have to merely due to the time it will take. 3) Use a BIOS I found posted elsewhere. No clue if it will work, but it may be worth a try.

Needless to say, it’s going to be an interesting experience. I will be sure to post the results for everyone since I know at least one pother person expressed interest in the x201 and my experiments could be handy.

Thanks again for all the recommendations.

So, I’m currently in the market for a new laptop. Little did I realize what a major headache it was going to be. I’ve had my current laptop (a first generation Macbook Pro) for the past 4.5 years. (Yes, I was one of those crazies desperately trying to get Gentoo onto a MBP way back when =P) And, given its age, and my current computing needs, I really do need an upgrade. However, I have been at it for a few days and so far I don’t have a *single* candidate for a new laptop. Now, I realize that I am very picky about a lot of things, however, I will say that the laptop companies don’t really make my life any easier. This is the bare minimum I am willing to accept:

1. Intel(i5) or AMD processor, at least dual core, quad if I can find it.
2. 4G DDR3 RAM
3. Dedicated graphics is preferred but not required. Bonus points if it has a dedicated card and an onboard card and can switch.
4. A solid state drive is a major plus, but isn’t required.
5. Atheros wireless card. *This* has been the bane of my existence.

Now, as if it isn’t bad enough that the majority of vendors seem to be using Broadcom *shudders*, NO ONE feels that it is a “tech spec” to list. I am SORRY, but saying 802.11 b/g/n tells me NOTHING. How hard would it be to put the bloody chipset there. But nooooooo, wouldn’t want that.

So, if anyone has a laptop that they love, that meets (or close to meets) requirements 1-3 and meets requirement 4, please, let me know. Cost is fairly flexible, but I’d like to keep it under $1200 if I can.

Thanks!

So, as some of you may have already noticed, I just recently joined the QA team. I’ve always enjoyed helping out with QA issues, mostly because it was one of the first things I got involved with given that my mentor was a member of the QA team (hwoarang). Add in that as far as I am concerned it is one of the single most vital projects as far as the distro is concerned and my motives become quite clear. Users, even the super savvy users that Gentoo usually attracts, have a distinct distaste for their OS breaking perpetually due to lack of QA.

So, since I started helping out with QA bugs (Which was right at the onset of the as-needed addition. Read there was a metric footon of LDFLAGS bugs), I seem to have this uncanny knack to pick bugs that also have some parallel compile issue embedded in them. Now, while these aren’t exactly a major QA issue (lets face it, it doesn’t really break anything), it is a concern for those of us with multi-core machines.

And that’s what? 90% of our user base…. (Disclaimer, 98% of all statistics are made up on the spot)

Given the fact that Gentoo is a completely from source distro, that most of us use a multi-core machine, and that some of use distcc to further the parallelization of that task, a complete lack of respect for makes jobserver is just STUPID.

Why the big deal? Lets say you have a four core machine. If either of the following two occur, you’re using ONE of the four. The other 3 sit around and do the other misc. tasks. Great use of your processor.. NOT!

We see this in a lot of ways, and I’ll go into them more in depth in a minute.

1) You’re compiling away, and at the end you see: warning: jobserver unavailable: using -j1. Add `+’ to parent make rule.

2) An ebuild uses -j1 with emake in src_compile / src_install. This, you don’t get a warning for from portage, but you DO from repoman if you’re the one working on the ebuild.

So, why do these happen?

#1 usually happens with poorly written Makefiles. Usually it’s that the Makefile uses ‘make’ in every (or some) lines instead of $(MAKE). How do we fix it? Well, the quick and easy way is usually to put:

sed -i -e ‘s/make/$(MAKE)/g’ Makefile (possibly more Makefiles if the project uses more) || die “Sed failed”

in src_prepare.

In around 9 of 10 cases, this fixes it completely. Sure, sometimes it’s more complicated. But why not fix it? The average user (someone with MAKEOPTS=”-jx” where x >= 2) will notice a HUGE difference in compile times.

#2 is a bit harder. Usually if an ebuild has an explicit emake -j1 it is because there is a known parallel-compilation issue. I wrote about this here. Usually what goes on is that the Makefile has a mistake where one of the targets has (a) missing dependencies (dependency). This can be a major headache to track down, and depending on how convoluted the build system is (and they get pretty sketchy from time to time) sometimes it really is not worth the several hours it might take to find and fix it. However, in some cases (like the ones I present in the link) they are fairly easy. And as such, I think it really benefits our users (and developers) to fix these if possible.

As such, I’ve opened a bug tracking primarily number 1 (but also perhaps number 2 if we can get there) since it is usually one of the easier to fix. As to number two, I am trying to find them in the crypto herd and fix those. Given that it might take an hour or so to fix, but spread over all our users / developers it may save many hours of compilation time, it seems like a good investment to me. Don’t you think?

Well, Jeremy requested screenshots, and now that I had something to take a screenshot of besides me shelling into my DX and chrooting, here they are!

This is lxde + icewm + tightvnc[server]. Not sure what I’m going to do next, but I figure people might enjoy this!

Soo a few days ago, I saw [1] and thought.. damn.. I want Gentoo on my DX. And so begins my saga.

Gentoo had been put on a Droid 1 before with little issue. So getting onto a Droid X should be possible. Right? As it turns out, yes! Now, the guide presented in [1] is not actually 100% the same as the procedure for the Droid X. So in an effort to make the road easier for someone else, I’m going to outline where I deviated. (NOTE: Must be rooted… duh!)

1) The qemu-user stuff is all correctish. It caused some headaches for me. Namely some “qemu: unknown syscall **” errors. So I put the chroot onto the device sooner than them.

2) I also setup and used distcc pretty heavily. I use it a lot to begin with, so it wasn’t too hard. Pitfall here was: make sure to use: “crossdev -S amrv7a-unknown-linux-gnueabi”. The -S forces stable, which is what your device and your chroot are using. It only makes sense the x-compiling device do the same.. (If you don’t it errors… a lot. Mostly on linking)

3) Now the real deviation. First off, make the image ext3. This is critical. The Droid X kernel is *not* xfs aware. You can then follow [1] up until the image is copied to the device. Now, for more differences. First off, /dev/loopX doesn’t exist on the DX in the same locations. The loopbacks are under /dev/block/loopX. This means running mount -o loop ** will not work. It’s going to try to use /dev/loop0 which doesn’t exit.. Also the loops in /dev/block are all *in use* so you can’t use those either.

To get around all of this, all you need to do is `mknod /dev/block/loop8 b 7 8` since loop8 should be the next available. Now you can do `losetup /dev/block/loop8 /path/to/image` followed by `mount -t ext3 -o loop /dev/block/loop8 /path/to/mount` and you have your mounted chroot!

Now what?

Well, me being the crypto geek that I am, was hellbent on getting cryptsetup working as part of the normal android OS (not in the chroot). And here is where Gentoo really really shines. Since we compile it all, you’re compiling it on the processor you’re using. Compile cryptsetup with USE=”-dynamic”, exit the chroot, copy it to /system/xbin and poof. Working encrypted images on the DX. (Same procedure as above. Create a blank file of whatever size you want, bind it to a new loopback device (mknod /dev/block/loop9 b 7 9), cryptsetup that loopback device, mkfs.ext2 /dev/mapper/yourdev, tune2fs -j /dev/mapper/yourdev, mount -t ext3 /dev/mapper/yourdev)

What else can we do?

Well, I’m looking into getting a working hostapd binary into the main android OS. Translation, free wireless AP that can use WPA2! Not to mention, you can add in any pentesting stuff you’d like. See [2] if you’re interested in that. A fellow Gentoo dev has already explored that route! =) The sky’s the limit. I’m working on getting a window manager and vnc setup. (mostly because I can)

Well, that’s most of the story. Feel free to post your ideas / suggestions. I can try to make my image available if people will use it. It’s still very bare-bones so to speak, so you wouldn’t have to jump through all the hoops I did to get it working =)

[1] http://howto.ccroms.net/howto/gentoo

[2] http://www.ribadeohacklab.com.ar/drupal/news/running-full-pentesting-environment-your-android-phone

Well, I finally got off my lazy rear and started working on some new fun for TrueCrypt. I’ve been intending to both install it and get familiar with it for some time. However, with the holidays, work, and finally moving into a permanent apartment, I’ve been a weeeee bit busy to say the least. Now, while the first two are still going on, I’m done with number 3, so I’ve recovered some of my free time. Translation: TrueCrypt is finally installed on my systems and I’ve started playing with it. Interestingly enough, I do kind of like it. It certainly has some serious overlap with cryptsetup, however, as was pointed out to me on a number of occasions, it serves others purposes as well. Admittedly, I’m impresses with some of it. So, what have I been up to?

Most notable of this is Bug 302170. As it was, if a baselayout-2 user didn’t remove their TC mappings before shutting down, the files ystem it resided on (ex. Container in /home) would not be able to unmount. Obviously this was a bit of a headache. So I finally got to working on a baselayout-2 init script for TC. It’s *very very* basic at the moment, and I would very much like to expand what it can do. As of right now, all it does is run the addon to run truecrypt -d. I would eventually like to have something along the lines of what we have with dm-crypt. A config file that can tell the init script to mount other partitions / containers at boot. And perhaps other things. Suggestions are welcome!

For those of you using TrueCrypt – Please get in there and test the new ebuild 7.0a-r2. If you encounter any problems with the new init script (Baselayout-2 users only), please get a bug open or an e-mail to me out asap! Thanks!

I don’t know when I’ll really get to working on the more advanced functionality. Hopefully soon. When i do, the bleeding edge of it will be available in my overlay. Again, anyone who is a bit on the daring edge and wants to help, it is most appreciated. If you’re in that category:

layman -s c1pher
echo "app-crypt/truecrypt" >> /etc/portage/package.unmask
echo "app-crypt/truecrypt **" >> /etc/portage/package.keywords

Do a dance!

I need all the help I can get with some of this, so please, if you have the time and know how, lend the hand =)

What else is going on with TrueCrypt?

I’m in the middle of trying to get “friendly” with upstream so to speak. I’m in the midst of trying to gain both access to their developer code base and their bug tracker. I figure if we want this working nicely in *nix, having a *nix developer upstream will probably be a bit of a hand. It also means I’ll be able to do something with live ebuilds / release candidate ebuilds. Translation: Fewer unknown bugs when we version bump.

Last, but not least, I’m aiming to finally get this into stable. I have *zero* clue if I will succeed with this, but I want to try. Once again, *please* run 7.0a-r2 and file any bugs you have. As of right now, I know of one bug that will prevent stabilization, however, I have been… pushy.. yea pushy =P, with upstream to try to get a fix for this. With some luck, I’ll be able to put it in =)

As always, if you have other ideas / suggestions / complaints feel free to comment or shoot me an e-mail. I’m all ears!

I will admit I didn’t realize there was as much use of TrueCrypt on Gentoo as I’ve come to find out there is.  This is good news in my opinion, and makes me feel a little bit better about sinking time into it. As long as people are making use of it, I don’t mind investing the effort =)

I’m a big fan of keeping users in the loop so as I make progress with TrueCrypt (and a lot of the other crypto packages) I will post here. Which leads me to my first post of this kind.

TrueCrypt is *no longer* fetch restricted. This accomplishes two major things. First, it means you don’t have to jump through extra hoops to install it, but second, and much much more importantly, we now have the source files on the Gentoo Mirrors. Translation, even when upstream bumps up to a new version, we can still provide the old one. Meaning if something breaks inadvertently (which just happens. It’s nothing against the TrueCrypt guys. I break things constantly =P) people aren’t screwed so to speak. It also means I can try to keep this package as stable as possible. Good news all around if you ask me.

As to the dm-crypt requests, I’m not doing anything super fancy in regard to Steganography, but I like to think my methods add an extra little but of… fun =) Given the demand, I will gladly write up what I’ve done and post it here. It will probably be a few days until I get around to this (more probably next weekend), but check back. Hopefully people will be able to make use of it.

If there are other Crypto things you’d like to see in Gentoo, feel free to shoot me an e-mail / prod me on IRC. I’m more than willing to see what I can do!

TrueCrypt has actually absorbed a fair bit of my time as a crypto dev. It’s amazing the number of issues that existed, and still exist, with that particular package. Worse off, I have a fairly strong understanding of the issues that can arise from a package like TrueCrypt breaking. All of the data on that encrypted partition is inaccessible, and depending on how long the fix takes, can be so for a while. This is an obvious problem.

So, this brings me to a few points. First off, *every* single one of my machines (save the router which has no actual information on it), has at least one encrypted partition. In the case of Feynamn (server), there are 4 storage partitions. Root is unencrypted. For my laptop, *every* partition is encrypted. The catch to this is, I *do not* use TrueCrypt. Why? Because as far as I am concerned there are *better* Linux solutions. See dm-crypt / cryptsetup. Though, I will confess, my method of using cryptsetup is fairly more advanced than most. Mine involved writing my own custom init script, amongst other things, to get it up and working the way I wanted.

*I am more than willing to provide this solution in a public format if there is demand for it. So if this is something you’d like to see, post below. If there is enough demand, I will gladly write it up here. I just don’t want to waste my time if people aren’t interested*

Now, back to TrueCrypt. Why do people use it then if there are better solutions? Because it’s cross-platform. However, their dev team can be a little less than co-operative from time to time, which causes headaches for us. For one, they refuse to provide us with download links for prior source versions, citing that they have security bugs. This is fine, except for when the new version breaks something and the old one is no longer possible. It also means that every new release, since our ebuild is fetch restricted, the older versioned ebuild is *USELESS* unless the user has the tarball stored. They also obfuscate their source tarball download link to “encourage” pointing users to the website to download it there. For a binary based distro, this isn’t an issue. They get their binary working, and distribute that. For us, this is a problem. And it’s one I plan on addressing hopefully today or tomorrow. And in theory, it should address issue number one since the downloads will be under Gentoo’s control. Hopefully I will get this working.

Whats the moral of all of this? I *do NOT* use TrueCrypt. So my ability to really test it is hindered. I am greatly encouraging TrueCrypt users to test my work and keep me in the loop as to how it’s working. I would like to keep this solution viable on Gentoo if I can. My favorite part of Gentoo is the options and the flexibility. I want to keep it that way.

But I NEED your help. Feel free to contact me by e-mail. I would love to hear your comments and concerns.

To those of you who don’t actually need to use TrueCrypt for OS compatibility (which by the way, there are ways to access dm-crypt partitions in M$ and OSX, but I think TrueCrypt is probably easier), I encourage you to check out dm-crypt. It’s more flexible, has a much less restrictive license, and is much better supported.

Well, I’ve finally taken care of becoming one of the elusive Sunrise Developers. Despite the fact that my top interest will always be the crypto herd, I have a soft spot for Sunrise. It’s a great way to get users involved with Gentoo in a fairly “painless” way for all parties involved. It allows users to “contribute” their ebuild to Gentoo without having to have them sitting in their own personal overlay that the grand majority of people will overlook. Add in that the overlay itself is monitored by a set of developers to attempt to maintain a certain standard of QA and everyone wins. Interestingly, many of the ebuilds in the Sunrise overlay are actually better written than some of the ones in the main tree, thanks to the process users have to go through before they can commit them. Sound like something you’d like to be involved in? Check it out! [1]

On the Gentoo and the Matrix note, a friend of mine sent me this image a day or two ago and I got a kick out of it, so I thought I’d share! Thanks Andy! Enjoy!

Hello Gentoo World!
So, I want to take the time to introduce myself. Some of you may already have seen some of my contributions as I’ve been active with Gentoo from a User perspective for the past couple months (mostly Bug Squashing and Sunrise). For everyone else (and even for you folks who might recognize me) I’d like to say hi!

I am very proud to say that I’ve been a Gentoo user since around ’02; and despite the frustrations most of us run in to from time to time, I can honestly say I’ve loved using it. It is due to this that I am very excited to be able to help out and contribute some of what I know and can do back to the community.

So, what am I going to be doing? My initial goal is to help out with the crypto herd (which has a good few open bugs I’d like to address) and the x86 team. In the near future, I hope to be able to get involved with the Sunrise Project in a formal sense. However, for the moment, I really would like to tame the crypto bug queue before I get further involved. Better to take the time to do a couple things right than to do a lot of things with less quality.

And now, back to it. I look forward to working with everyone!

It’s been a long while I know. So I decided now was a good time for a mini-update. I have been jumping through all of the requirements to try to secure myself employment. Between the 180+ hours on a presentation, and the endless miles of paperwork and packing… I’ve been busy.

Now, while all of this has been going on, I haven’t had a ton of time to throw into Gentoo the way I had been. However, the recruiters finally did get around to processing my app, and as of this morning, I am officially becoming a full blown developer. So, once things settle down, and I find a suitable apartment, I can finally get some of the bugs for the crypto herd closed out. Good times!

Is real.

I find it majorly ironic that almost immediately after I post the “Parallel Build Fixing” I get a parallel build bug that makes me want to rip my hair out. What’s worse about it is the fact that the Makefile wasn’t really the issue (as far as determining what was causing the failure). It was the amount of warnings the code generated.

I kid you not. Around 5 – 10 warnings for EVERY SINGLE SOURCE FILE.

The error actually occurred so far up it took me about 3 hours to find the stupid thing because I ended up chasing down 3 dead-ends from the rest of the code. (That displayed farther into the build, leading me to believe they were the *only* error.)

So, what was the actual root of the problem?

For reference, @TARGET@ = NOTSall

all:    build-sub @TARGET@
Sall:           $(PROG) $(PROGSERVER) $(PROGINDEX)
agrep:          $(PROGAGREP)
NOTSall:        $(NOTSPROG) $(NOTSPROGSERVER)

build-sub:
for d in $(SUBDIRS) ; do \
( cd $$d; $(MAKE) ); \
done

The key was that NOTSall needs a library compiled by one of the sub-directories. The fix is actually fairly simple, once you manage to find the stupid problem.

all:    build-sub
Sall:           $(PROG) $(PROGSERVER) $(PROGINDEX)
agrep:          $(PROGAGREP)
NOTSall:        $(NOTSPROG) $(NOTSPROGSERVER)

build-sub:
for d in $(SUBDIRS) ; do \
( cd $$d; $(MAKE) ); \
done
$(MAKE) @TARGET@

Now, that it is located there, it is guaranteed to be built after the sub-directories. Problem gone.

Hard? No. Not really. Painful? HELL YES!

One of the more common QA issues I’ve been running into is packages that fail to build with multiple jobs, aka parallel make bugs. Although they aren’t always the easiest to test for, they are usually fairly straightforward to fix. Unfortunately, in a lot of cases, the “fix” seems to be just use make -j1. Personally, I don’t like this. I have a quad-core desktop, and to see a machine that is capable of compiling 5 things at once doing one thing at a time makes me rather upset.

How does one fix these issues? It varries. Most of the time, it is a simple case of a dependency being missing from one of the make targets. For exmaple:

Makefile:
…
fish.a: mylib.o fakelib.o
$(CC) $(LDFLAGS) fish.o mylib.o fakelib.o -o fish.a

catch_fish: catch.o
$(CC) $(LDFLAGS) fish.a catch.o -o catch_fish
…

With that makefile if you are building with multiple jobs, it is quite possible it will try to build catch_fish either while fish.a is compiling, or before fish.a compiles. If this happens, make generally get very angry and spits out all kinds of nasty errors. All you need to do is make sure catch_fish depends on fish.a. A targets dependency list is everything after the colon. So our fixed makefile would look like:

…
fish.a: mylib.o fakelib.o
$(CC) $(LDFLAGS) fish.o mylib.o fakelib.o -o fish.a

catch_fish: catch.o fish.a
$(CC) $(LDFLAGS) fish.a catch.o -o catch_fish
…

Take for example bug 326385. We end up with the angry make error:
./libnettle.so: file not recognized: File truncated

Translation: libnettle.so isn’t _done_ compiling, so whatever is trying to link against it is very upset. When we look closer at the bug report, we can see that libhotweed.so is using -lnettle (look at the **’s). However, this looks for libnettle.so which starts compiling *after* libhotweed in this case. (It could be before and just not have finished as well):

x86_64-pc-linux-gnu-gcc -Wl,-O1 -Wl,–hash-style=gnu -shared
-Wl,-soname=libhogweed.so.1 sexp.po sexp-format.po sexp-transport.po
sexp-transport-format.po bignum.po bignum-next-prime.po bignum-random.po
sexp2bignum.po pkcs1.po pkcs1-rsa-md5.po pkcs1-rsa-sha1.po pkcs1-rsa-sha256.po
rsa.po rsa-sign.po rsa-verify.po rsa-md5-sign.po rsa-md5-verify.po
rsa-sha1-sign.po rsa-sha1-verify.po rsa-sha256-sign.po rsa-sha256-verify.po
rsa-encrypt.po rsa-decrypt.po rsa-keygen.po rsa-compat.po rsa2sexp.po
sexp2rsa.po dsa.po dsa-sign.po dsa-verify.po dsa-keygen.po sexp2dsa.po
pgp-encode.po rsa2openpgp.po der-iterator.po der2rsa.po -o libhogweed.so -L.
******-lnettle****** -lgmp

Here is our issue. Now, we can go back to the Makefile, psych out which make target it is. In this case it is:
$(LIBHOGWEED_FORLINK): $(hogweed_PURE_OBJS)
$(LIBHOGWEED_LINK) $(hogweed_PURE_OBJS) -o $@ $(LIBHOGWEED_LIBS)
-mkdir .lib 2>/dev/null
[ -z "$(LIBHOGWEED_SONAME)" ] || (cd .lib \
&& ln -sf ../$(LIBHOGWEED_FORLINK) $(LIBHOGWEED_SONAME))

Upon inspection of $(hogweed_PURE_OBJS), we can see that it doesn’t include libnettle.so. More importantly, when we go to config.make (it’s imported at the top) we can see that this is where -lnettle is specified. As such, the dependency list for $(LIBHOGWEED_FORLINK) (everything that comes after the colon is the dependency list) should *include* nettle.so. Since the Makefile is generated from Makefile.in we go and fix that there. This is a bit more of a headache as everything is more generic, but it’s usually easy enough to figure out.
Here is the generic LIBHOGWEED_FORLINK rule in Makefile.in. Not that different thankfully:
$(LIBHOGWEED_FORLINK): $(hogweed_PURE_OBJS)

We now know it needs to also depend on LIBNETTLE_FORLINK so we add that to the dependency to get:
$(LIBHOGWEED_FORLINK): $(hogweed_PURE_OBJS) $(LIBNETTLE_FORLINK)

Generate the diff and you have your patch. Result for this was:
— Makefile.in    2009-06-08 14:27:11.000000000 -0400
+++ Makefile.nettle.in    2010-08-09 19:58:19.000000000 -0400
@@ -148,7 +148,7 @@
[ -z "$(LIBNETTLE_SONAME)" ] || (cd .lib \
&& ln -sf ../$(LIBNETTLE_FORLINK) $(LIBNETTLE_SONAME))

-$(LIBHOGWEED_FORLINK): $(hogweed_PURE_OBJS)
+$(LIBHOGWEED_FORLINK): $(hogweed_PURE_OBJS) $(LIBNETTLE_FORLINK)
$(LIBHOGWEED_LINK) $(hogweed_PURE_OBJS) -o $@ $(LIBHOGWEED_LIBS)
-mkdir .lib 2>/dev/null
[ -z "$(LIBHOGWEED_SONAME)" ] || (cd .lib \

And now you’re done with that issue. The next time it tries to build libhogweed.so it will make sure libnettle.so is done BEFORE it even starts. No more issue, and no slowing down the build for people with lots of cores.

Two more packages have made their way into Sunrise. Namely, dev-ruby/PlugMan and net-analyzer/hostmap. Working with the hostmap guys was  very smooth. Ran into an issue with their package naming, and got it fixed almost the same day. Now that I’ve dealt with a couple Ruby ebuilds, as much as I like how they handled the versioning issue,  I don’t like having to constantly learn new eclasses for every single thing. I really wish there was a more global understanding for how to deal with some of the problems. The Python guys do it one way, the Ruby guys another. It makes for a lot of work for the more general devs.  But so it goes I guess. Thankfully for me, most of the crypto packages are plain old C and C++. So as long as I can avoid the pesky bash build-systems, I should be fine.

To start it blunt, they can die in a raging car fire.

In all seriousness however, it is a major headache for anyone who is trying to figure out 1) how it working, and 2) how to make it play nice with others. Autotools exist. USE THEM.

I spent no less than 7 hours trying to get the ebuild for dev-libs/cryptlib to play nice with others and 1) respect USE flags correctly 2) actually allow for more than 1 job at a time. BOTH of these issues would have been completely moot if upstream used a sane build system. Instead, the default make target builds 2 things, then spawns off and calls a bash script (which calls other bash scripts which use a whole bunch of automagic), which after a whole lot of other scripts being called, eventually goes back to the main directory and calls make AGAIN… with the proper targets.

*sigh* It’s all better now though. Hopefully, it will stay that way.

I’m getting ready to move back to Albany. To do this, I will have to do the unthinkable. Turn off all the computers that are currently running happily in my room. One of these computers, my everything-in-a-box server (git, cvs, samba, openvpn, and everything else I find useful), Feynman, has been up since I built him and configured him last summer. I got the idea to look at hist uptime today, it is 09:51:19 up 291 days, 23:21,  6 users,  load average: 0.39, 0.28, 0.34

And now I’m going to have to turn him off. Only 74 days to a 1 year uptime. DRAT! Whats sad is that I would consider building a little battery setup to keep him on in the car to keep my uptime. But, truth be told though, he could use the reboot. Whats funny to think is that 2.6.30 was bleeding edge when I got him up and running. I think it may be time for an upgrade.

Well, it has been busy. The New Dev bug is officially filed, so the excitement is about to begin on that front. I’ve been learning a lot though, mostly thanks to Hwoarang.

Now that all the quizzes are done, at least to Hwo’s satisfaction, I’ve had some time to actually start attacking the rather large queue of Crypto bugs, and in doing so, I’ve already come up with a list of things I hate to see in a Makefile =P. I did actually manage to solve a couple of the problems, and some of the QA ones are already committed. However, the bulk of the work still has not been pushed into the Portage tree. Hopefully one of the Crypto dev’s will get around to that stuff soon.

Also, a note. Camping in tornadoes = exciting =D.

Well, I’ve been at it all day, but I finally sent my End Quiz off to Hwoarang to look at. I have to say, it was surprisingly more technical than the Ebuild Quiz, and actually took me a good deal longer to finish. Now, all I have left for the quizzes is any corrections that are necessary (which in the case of the End Quiz, I’m sure I have quite a bit to do =P) Time for a much earned nap.

Spent most of my morning going back over my Ebuild Quiz (as I anticipated) addressing the list of things Hwoarang though I should improve. As much as they aren’t particularly “fun” to do, I actually got a lot out of it. He was able to point out a good few things that I hadn’t really thought of that will probably be good to know in the future. I’m sure there will be more to correct / revisit in the near future =P

I also got myself a user overlay set up on git.overlays.gentoo.org. It’s mostly things that are already either in Sunrise or in the main tree already, but I still think it will prove useful.

In addition to that, I picked up another package to Proxy-Maintain (app-crypt/bsign) and discovered that the good old ROT13 implementation in app-crypt/rotix seems to have completely died upstream. It actually makes me kind of sad to see a Crypto project die. But oh well. Such is life.

Then I discovered this thanks to Fauli and managed to get myself distracted for a good couple hours messing around with it. I’m actually kind of impressed with it. It’s nice to see so many Open Source people around. I set myself up an account. Feel free to follow me! (c1pher)

Sleep now. I get to work on my End Quiz in the morning, which hopefully will prove just as “enlightening” as the first one.

After a good month of lurking around and helping out where I could, I finally found a Dev that is willing to Mentor me. Big thanks to Hwoarang for deciding to help me along! Thankfully, I took care of the Ebuild Quiz when I started arch testing, though I am sure I will have a good deal of that to fix.  However, thats only the first half the battle. Now I get to start working on my End Quiz, which if the Ebuild Quiz was any indications will take me close to 5 hours to do start to finish. Oh yay. And then I’m sure there will be plenty to fix on that too.  But, I honestly can’t complain. It’s nice to finally start moving towards becoming a Developer. =)

It took a few weeks, but one of my ebuilds finally hit the main Portage tree. It was actually a pretty nice feeling seeing it leave Sunrise and head for the real thing. I’m hoping to get another one of my packages, namely Ostinato, into Portage soon. Proxy-Maintaining should prove to be an experience.