About TODO Blog RSS Old blog Projects Gallery Notes

Thu, 19 Jul 2007

Distributed storage progress.

So far I completed networking processing rewrite, so that there are zero additional allocations during fast path (comapre to two in device mapper in the best case), although code itself is a bit ugly in places, so I will clean this up eventually. There is one possible allocation if fast path would end up sleeping in sending of receiving function, in that case new request is allocated from memory pool and queued to be processed by dedicated worker thread later (when socket is awakened by bottom half or sending route), this queue is protected via RCU.
I have not yet tested local device node, i.e. when storage contains local disk as one of its nodes, there is also not yet tested local export mode - this allows to export local disk to remote peers, right now I use userspace export daemon, which works with local files.

Hunting for tricky bug took most of the last couple of days - bug happend when huge block IO requests ended up covering multiple nodes. In device mapper it is fixed simply by allocating additional block IO request, but since I decided to remote additional block IO allocations completely, I needed to make dst processing engine flexible enough to allow such cases. The most tricky part was situation, when single bio_vec, i.e. a page, happend to be on the boundary so that parts of the page are on different nodes. Simply playing with size and offset of the page is not alowed, since XFS requres that fields to be untouched by the block layer (this is the only such restricting user though). Local node actually requires to allocate new BIO here, so in local hot path there might be aditional allocation, but there is no postponed requests though - I just queue new bio at the end of the queue for the specified device by calling generic_make_request().

I've started massive killing testing of the system - the whole night it will mount/umount, read/write, sync and perform filesystem rebuilding, if it will not crash or freeze and log of this crap will not contain broken data, I will consider remote transfer mode as ready. When it is, local node support and exporting testing will not take too much time (although I already said something similar and it took me about a week to move forward, but everything is good which ends up being good).

Because of this bug I did not move to development shop and thus did not complete ceiling paining, and thus still do not have wallpapers on the walls and cover on the floor, and the whole appartments developing process is in a hung state - every day I fall asleep in my hammock looking to tons of garbage, concrete and instruments around. But I will fix that too, it just requires a bit more time than I expected (actually it requires exactly those several days I expected, but only in case I do work in this problem, but so far I usually do something more interesting. I can pay for completing my loft, but that is not interesting variant)...

Stay tuned.

/devel/dst :: Link / Comments (0)