User:	apenwarr_advo
Date:	2010-01-06 18:55
Subject:	The Google Phone
Security:	Public

http://apenwarr.ca/log/?m=201001#07

...even Microsoft never was brazen enough to pull something like this. Even Microsoft had some tiny bit of shame. Google is a different beast altogether. They're like nothing anyone has ever seen in our business. Not only are they not ashamed - they think they're the good guys!

-- Fake Steve Jobs

Dot dot dot.

User:	apenwarr_advo
Date:	2010-01-03 14:39
Subject:	Dealing with "tumultuous" economic times
Security:	Public

http://apenwarr.ca/log/?m=201001#06

Canadian Economics Superhero Larry Smith has a new article out, addressing burning economic question questions such as:

• Why is Silicon Valley so shortsighted?
• Why do so many people write iPhone apps and so few write medical software?
• How to deal with a shortage of engineers.
• Volume of information vs. value of information.

	User:	ink_13
	Date:	2010-01-06 22:23
	Subject:	Mulligan
	Security:	Public

I wrote something for someone else that probably shouldn't be posted publicly (it concerns HR practices at my company), and it expended my writing energy for the night, so instead, we get this placeholder.

I broke two glasses yesterday in a fit of clumsiness. Today I went and bought four glasses (I decided I wanted more glasses). Then I had really excellent char siu fried rice that only cost $4, and I even had enough left over for lunch tomorrow. Three cheers for Kom Jug Yuen!

	User:	mixedchick6
	Date:	2010-01-05 22:55
	Subject:	New year
	Security:	Public

Happy 2010 everyone :)

	User:	ink_13
	Date:	2010-01-05 22:49
	Subject:	And then there were two
	Security:	Public

As if being 25% down wasn't annoying enough, now half of the elevators in my building are out of commission. Empirically, it seems travel times are up by 5 minutes until they get fixed.

I'd walk, but in order to have this view, one does have to be far off the ground, which makes elective stair use inconvenient at best.

	User:	jwz
	Date:	2010-01-05 13:50
	Subject:	I wish to see these films.
	Security:	Public
	Music:	Sheila E. -- The Glamorous Life

	User:	jwz
	Date:	2010-01-05 12:44
	Subject:	Well done, Kodak! The Goatses will continue until morale improves.
	Security:	Public
	Music:	Powerman 5000 -- An Eye Is Upon You

How to show everyone what's on your Kodak frame.

So much win:

• If you know or guess the MAC address of any Kodak wireless digital picture frame, you can extract the images that are displaying on it.

• You can also remotely reset the frame, meaning you can 0wnz0r it and change its image sources.

So Kodak has built an appliance for letting complete strangers (a) browse your family photos, and (b) beam shock porn directly into your living room! GOD BLESS AMERICA!

This all works because the appliances won't connect to (e.g.) Flickr directly, they only phone home to Kodak's server, which then proxies all of the requests. But at least they're using OAuth instead of making you type your Flickr password into Kodak's server. This is a little surprising, actually, given the tip-top job their security engineers did of designing the rest of the infrastructure of this product line.

I guess I ought to add a WebCollage source to generate random Kodak MAC addresses for use as an image source!

	User:	tamedknee
	Date:	2010-01-05 11:04
	Subject:	Yesterday
	Security:	Public

With a flight at 8:20 p.m. and given the added security measures/restrictions, I thought leaving the house at 5:00 p.m. would probably suffice, but wanted to leave at 4:00-ish to be prudent (especially with rush hour traffic) - we could get an early dinner along the way.

So approximately 12:15 p.m., my mom wanted to buy and upload an updated map for her GPS while I was still around. When it came to purchase the map, the total was in USD, at which point I was asked if my US credit card could be used to avoid paying an exchange rate. I agreed and headed downstairs to get my wallet... which turned out to be missing.

The previous night we had gone to the movies, and I knew I had had it there. We called the theater but nobody picked up. My mom drove to the theater to check; I decided I needed to pack for my trip back. My mom returned to tell me that the theater was not open until 3:15, and she didn't see anything near where we had parked.

We figured it would be best to treat it as lost/stolen. I spent the next hour on the phone cancelling bank cards and credit cards. On the advice of my uncle, we went to the police to file a report, and then to get a replacement drivers license (not that I actually drive here, but it's something I suppose I'll need eventually). We figured lining up for a new OHIP card would take too long. It was 3:30 after getting my temporary license. We decided to go to the theater. At the theater, there wasn't much staff to help me. I finally found the manager who was selling tickets at the concession stand. I was told that found wallets would be kept in the office and I'd have to wait until all patrons had been served before they could go check (reasonable - but slightly distressing when it rolled to 4:10). They did have my wallet. At 4:15, the manager came out with my wallet - nothing missing.

I had a cousin who was flying to New York at approximately the same time as me, they had left at 3:00 (she had also wanted to renew her visa); we received a call from my aunt telling us that the airport was chaotic and we should get there as soon as possible - which rushed our dinner plans. We got to the airport at 5:10 p.m. Lineups were long, but moved quickly and I got to my gate at 6:10. I found my cousin and we talked until our flights boarded. Apparently I had missed some chaos with the customs line; that they had eventually closed the customs line for a while. I recalled that while waiting to tag my luggage I had heard an announcement regarding customs to the US had reopened (wasn't sure what it meant at the time)

	User:	jwz
	Date:	2010-01-05 00:20
	Subject:	I, for one, welcome our new Hivemind Overlords.
	Security:	Public
	Music:	Ciccone Youth -- March of the Ciccone Robots

A couple of good stories I came across today:

• The Things by Peter Watts -- The Thing from the perspective of the alien.

• The Gentle Seduction by Marc Stiegler -- The Rapture of the Nerds from the perspective of a non-nerd.

	User:	jwz
	Date:	2010-01-05 00:12
	Subject:	hey, you got your Buttered Cat Effect in my Security Theatre!
	Security:	Public
	Music:	Bauhaus -- The Man with the X-Ray Eyes

What happens when the immovable object of terrorism meets the unstoppable force of kiddie porn?

Previously, previously.

	User:	ink_13
	Date:	2010-01-04 22:55
	Subject:	Thoughts on having a smartphone for 24 hours
	Security:	Public

"I have the Internet in my pocket!
...
Let's read some stuff!
...
Holy shit. I live in the future!"

	User:	jwz
	Date:	2010-01-04 01:29
	Subject:	dnalounge update
	Security:	Public
	Music:	Midnight Juggernauts -- Tombstone

DNA Lounge update, wherein we bend over and take it.

User:	apenwarr_advo
Date:	2010-01-05 12:44
Subject:	bup 0.01: It backs things up
Security:	Public

http://apenwarr.ca/log/?m=201001#04

I just spent a few days of my Christmas vacation writing a new program, bup.

bup is a program that backs things up. It's short for "backup." Can you believe that nobody else has named an open source program "bup" after all this time? Me neither. It also has almost no other meanings.

Despite its unassuming name, bup is pretty cool. To give you an idea of just how cool it is, I wrote you this poem:

Hmm. Did that help? Maybe prose is more useful after all.

Reasons bup is awesome

bup has a few advantages over other backup software:

• It uses a rolling checksum algorithm (similar to rsync) to split large files into chunks. The most useful result of this is you can backup huge virtual machine (VM) disk images, databases, and XML files incrementally, even though they're typically all in one huge file, and not use tons of disk space for multiple versions.
• It uses the packfile format from git, so you can access the stored data even if you don't like bup's user interface.
• Unlike git, it writes packfiles *directly* (instead of having a separate garbage collection / repacking stage) so it's fast even with gratuitously huge amounts of data.
• Data is "automagically" shared between incremental backups without having to know which backup is based on which other one - even if the backups are made from two different computers that don't even know about each other. You just tell bup to back stuff up, and it saves only the minimum amount of data needed.
• Even when a backup is incremental, you don't have to worry about restoring the full backup, then each of the incrementals in turn; an incremental backup *acts* as if it's a full backup, it just takes less disk space.
• It's written in python (with some C parts to make it faster) so it's easy to extend and maintain.

(The README actually has a more detailed example.)

Try making a remote backup:

 	tar -cvf - /etc | bup split -r myserver: -n my-etc -vv

 	bup join -r myserver: my-etc | tar -tf -

(On myserver) look at how much disk space your backup took:

 	du -s ~/.bup

 	tar -cvf - /etc | bup split -r myserver: -n my-etc -vv

Look how little extra space your second backup used on top of the first:

 	du -s ~/.bup

   	bup join -r myserver: local-etc~1 | tar -tf -

I have lots of plans for this lovely program, in the event that I actually get time to implement them. But if you think it's cool, please feel free to git clone it, hack away, and send some patches! Read the README for a list of some deficiencies in the current release.

I'm sure there are also more deficiencies that I don't know about, of course.

(Previous poetry-related adventures.)

Update (2010/01/05): Commentary at ycombinator news and reddit programming. To answer the most common question: it's different from most of those other apps you mention because: a) bup backs up really huge files rather than silently ignoring them or running out of memory; b) bup is a backend, not a GUI, while most of those apps are GUIs (which could use bup as a backend if they wanted); c) bup stores its backups in big packfiles, rather than a one-file-per-file model, and thus can be much faster (but 0.01 isn't optimized yet).

Update 2 (2010/01/05): By popular demand (well, nonzero demand from the populace, anyway), I've created a mailing list. You can subscribe by sending an email to bup-list+subscribe@googlegroups.com (note the weird + character in the email address).

	User:	ink_13
	Date:	2010-01-03 22:34
	Subject:	Backlog clearing
	Security:	Public

Extremely short music reviews: Lady Gaga edition.

Stefani Germanotta is the most talented thing to hit pop music since Christina Aguilera, and is probably a better musician (Exhibit A). She has figured it out, not just from the correct use of the hook, the other hook, and the contrapunctuality to be gained from their interplay (i.e., how to write perfect pop music), but more importantly how to play and exploit the fame game. Her entire celebrity is a masterpiece, and I'm in awe.

She is playing an elaborate joke on the entertainment industry (which is not a secret), and they're eating it up. "She is," as one Reddit commentator put it, "to pop music what Stephen Colbert is to conservatism."

The music? On the surface, it's vaguely-Eurotrash-imitation dance-pop, but my enjoyment went up exponentially when I realised there's a serious musician behind all of it who is quietly teasing and I started listening more closely. The polyphony alone (probably my favourite thing in music — if it ain't baroque...) is astounding. She intends its excesses to be ridiculous. I can hear an impish grin in Bad Romance. It's supposed to be like that.

I listened to her albums hoping for something fluffy and bad I could pithily dismiss while trying to see which of the tropes she was exploiting to be the flavour of the week. I instead came out deeply impressed and appreciative. I don't know if she still does club shows, but if she does, I'm distinctly interested in attending one.

User:	apenwarr_advo
Date:	2010-01-03 06:37
Subject:	Version control of really huge files
Security:	Public

http://apenwarr.ca/log/?m=200910#04

So let's say you've got a database with a 100k rows of 1k bytes each. That comes to about 100 megs, which is a pretty small database by modern standards.

Now let's say you want to store the dumps of that database in a version control system of some sort. 100 megs is a pretty huge file by the standards of version control software. Even if you've only changed one row, some VCS programs will upload the entire new version to the server, then do the checksumming on the server side. (I'm not sure of the exact case with svn, but I'm sure it will re-upload the whole file if you check it into a newly-created branch or as a new file, even if some other branch already has a similar file.) Alternatively, git will be reasonably efficient on the wire, but only after it slurps up mind-boggling amounts of RAM trying to create a multi-level xdelta of various revisions of the file (and to do that, it needs to load multiple revisions into memory at once). It also needs you to have the complete history of all prior backups on the computer doing the upload, which is kind of silly.

Neither of those alternatives is really very good. What's a better system?

Well, rsync is a system that works pretty well for syncing small changes to giant files. It uses a rolling checksum to figure out which chunks of the giant file need to be transferred, then sends only those chunks. Like magic, this works even if the sender doesn't have the old version of the file.

Unfortunately, rsync isn't really perfect for our purposes either. First of all, it isn't really a version control system. If you want to store multiple revisions of the file, you have to make multiple copies, which is wasteful, or xdelta them, which is tedious (and potentially slow to reassemble, and makes it hard to prune intermediate versions), or check them into git, which will still melt down because your files are too big. Plus rsync really can't handle file renames properly - at all.

Okay, what about another idea: let's split the file into chunks, and check each of those blocks into git separately. Then git's delta compression won't have too much to chew on at a time, and we only have to send modified blocks...

Yes! Now we're getting somewhere. Just one catch: what happens if some bytes get inserted or removed in the middle of a file? Remember, this is a database dump: it's plaintext. If you're splitting the file into equal-sized chunks, every chunk boundary after the changed data will be different, so every chunk will have changed.

This sounds similar to the rsync+gzip problem. rsync really sucks by default on .tar.gz files, because if a single byte changes, every compressed byte after that will be different. To solve this problem, they introduced gzip --rsyncable, which uses a clever algorithm to "resync" the gzip bytestream every so often. And it works! tar.gz files compressed with --rsyncable change only a little if the uncompressed data changes only a little, so rsync goes fast. But how do they do it?

Here's how it works: gzip keeps a rolling checksum of the last, say, 32 bytes of the input file. (I haven't actually looked at what window size gzip uses.) If the last n bits of that checksum are all 0, which happens, on average, every 2^n bytes or so, then toss out the gzip dictionary and restart the compression as if that were the beginning of the file. Using this method, a chunk ends every time we see a conforming 32-byte sequence, no matter what bytes came before it.

So here's my trick: instead of doing this algorithm in gzip, I just do it myself in a standalone program. Then I write each chunk to a file, and create an index file that simply lists the filenames of the required chunks (in order). Naturally, I name each chunk after its SHA1 hash, so we get deduplication for free. (If we create the same chunk twice, it'll have the same name, so it doesn't cost us any space.)

...and to be honest, I got a little lazy when it came to creating the chunks, so I just piped them straight to git hash-object --stdin -w, which stores and compresses the objects and prints out the resulting hash codes.

An extremely preliminary experimental proof-of-concept implentation of this file splitting algorithm is on github. It works! My implementation is horrendously slow, but it will be easy to speed up; I just wrote it as naively as possible while I was waiting for the laundry to finish.

Future Work

For our purposes at EQL Data, it would be extremely cool to have the chunking algorithm split based only on primary key text, not the rest of the row. We'd also name each file based on the first primary key in the file. That way, typical chunks will tend to have the same set of rows in them, and git's normal xdelta stuff (now dealing with a bunch of small files instead of one huge one) would be super-efficient.

It would also be entertaining to add this sort of chunking directly into git, so that it could handle huge files without barfing. That would require some changes to the git object store and maybe the protocol, though, so it's not to be taken lightly.

And while we're dreaming, this technique would also be hugely beneficial to a distributed filesystem that only wants to download some revisions, rather than all of them. git's current delta compression works great if you always want the complete history, but that's not so fantastic if your full history is a terabyte and one commit is 100 GB. A distributed filesystem is going to have to be able to handle sparse histories, and this chunking could help.

Prior Art

I came up with this scheme myself, obviously heavily influenced by git and rsync. Naturally, once I knew the right keywords to search for, it turned out that the same chunking algorithm has already been done: A Low-Bandwidth Network Filesystem. (The filesystem itself looks like a bit of a dead end. But they chunk the files the same way I did and save themselves a lot of bandwidth by doing so.)

	User:	jwz
	Date:	2010-01-02 18:43
	Subject:	videos
	Security:	Public
	Music:	as noted

	User:	jwz
	Date:	2010-01-02 13:39
	Subject:	Today in Torture Phallus news:
	Security:	Public
	Music:	Oxygiene 23 -- Adonis

Ducks fight the battle of the sexes in their genitals

The male duck's penis is spiral-shaped: like a corkscrew, it twists in a counter-clockwise direction so that sperm will target the oviduct on the female's left-hand side. In almost all birds only the left ovary is functional, but in a 2007 study, Brennan and colleagues noticed that in ducks the female's vagina twists in the opposite direction.

Brennan thinks that, while the males are evolving long and flexible penises to help them force copulations, the females are using their complex vaginal anatomy to take back control over which sperm fertilises their eggs. When a female wants to mate with her chosen partner, she can make the process easier by relaxing the muscles around the vagina entrance.

Brennan's team also timed the male's penis eversion, which took a mere one-third of a second - around 60 times faster than was previously thought (see video above). "This definitely gives the males a mechanism by which they can copulate," says Brennan, who was taken aback by the speed. "To be totally honest, I'm still in shock," she says.

Previously, previously, previously.

	User:	jwz
	Date:	2010-01-02 13:24
	Subject:	how deep does this navel go?
	Security:	Public
	Music:	The Faint -- The Geeks Were Right

There are 15,740 social media gurus on Twitter.

445 social media gurus
12 viral marketing gurus
340 internet marketing gurus
20 new media marketing gurus
41 digital marketing gurus
68 social media stars
8112 social media marketers
2091 social media consultants
807 social media experts
931 social media strategists
576 social media agencies
79 social media ninjas
626 social media companies
271 social media marketing specialists
560 internet marketing specialists
154 internet marketing agencies
401 digital marketing agencies
206 digital marketing strategists

	User:	dkchisho
	Date:	2010-01-02 15:28
	Subject:
	Security:	Public

2009 was a pretty awesome year in many ways. Got engaged, got married, to name a couple of significant ones. :) I also lost two grandparents, and had a really rough time at my job. But mostly the year was good.

In 2010 I'm looking forward to continuing to enjoy married life, getting re-energized at work, and possibly other significant life events - such as deciding to do a PhD if I do get the funding I applied for. I don't think I kept any of my resolutions from last year (I'm not even looking at them) but I will continue to try to be better in various aspects of life.

Only teaching one class next term will mean I have more time to do personal things, so I'm planning to get back into cross-stitch. (I don't think I touched any of them in the entirety of 2009... just brought them around with me various places and never did anything on them.) I want to make a stained glass window for our front door, too, possibly to match the ones I made for my parents last Christmas.

My husband is sick in bed right now, after the worst holiday I can possibly imagine (involving three trips to the dentist and more painkillers in two weeks than I have ever seen him take in 5 years combined) but somehow he still manages to be wonderful and supportive and amazing. Here's hoping he gets better soon!

	User:	swestrup
	Date:	2010-01-02 10:09
	Subject:	Call it a Learning Experience.
	Security:	Public

So, I had a great time at the New Years Eve party at sjfries and inadvertently got way too drunk. I don't often drink to excess and I don't think I've ever drunk enough before to feel that bad. And I'm not talking hangover. I realized when the party ended that my face felt like it was trying to crawl off my skull. All the muscles in it were pulling in different directions. It was MOST unpleasant.

I drank a large amount of water and then crashed in the spare bedroom for the night, but I woke up 5 hours later with one whopper of a hangover. I actually didn't have much trouble with headache or light sensitivity (I guess I wasn't too dehydrated because of the water) but was incredibly nauseous. I probably threw up about 5 times. Luckily I'm a good guest and never missed the toilet bowl.

So, I basically spent the whole day laying down in the guest bedroom, dozing and drinking water. By 6:00 pm I was well enough to have Linda come and pick me up, and went home. I tried to eat something, but my stomach was still so tender that all I managed to was a banana and some dry toast, washed down with ginger ale (which is good for nausea). I puttered around a bit until 10:00 pm hoping that I would start feeling better, but when it became clear that I needed another full nights sleep to recover, I went to bed. And now I'm awake, feeling fine.

So, call it a learning experience. I now know of a point of inebriation that I NEVER want to get to again. I knew that it would inevitably happen when I started drinking. In order to know my limits, I had to exceed them at some point. Well, now I know.