[LUNI] in search of filesystem wisdom

[Martin Maney]

On Sat, Sep 08, 2007 at 03:57:59PM -0500, Kenneth P. Stox wrote:
> UFS2 does not have on disk indexing, nor does it use B+trees for

Thanks, I've been having a hard time finding a simple, straightforward
statement like this online.  I have come across plenty of sotries about
the hassles of having so many files in one place, but mostly they're
about the shell and tools' limits.  Did find one that was a little
scary (both relevant and pretty recent):

  http://www.nabble.com/Slow-locking-UFS-VFS-in-6.1--t2224669.html

> Currently, you would be better off with XFS or Reiser on a Linux machine

Interesting - I knew XFS was s'posed to be mighty good for working with
huge files, don't hear so much about it being good for huge numbers of
[not so huge] ones.

> the old work-around, ./first-letter/second-letter/lots_of_files_here
> would be your best approach.
> 
> Also, last I looked, although some filesystems can handle it, many shell
> scripts and some tools will break when used in/on directories with huge
> numbers of files. Keeping file counts below 1000/directory would be in
> your best interests anyway.

As I said, if I were starting from scratch, I'd have done it
differently.  :-/  The shell/tool limits really aren't a big issue -
this is one of those cases where the now-departed developer figured
there was no point in stuffing the images into the database with the
metadata when he had a nice filesystem sitting there.  Since this also
lets them be directly accessed by a high-performance web server, that's
a perfectly reasonable choice here.  But he clearly wasn't thinking
about scaling, alas.

Guess we'll have to invest in some re-engineering.  This would have
been so much easier if it had been done right at first.

-- 
If God did not exist, it would be necessary
for us to invent Him.  --Voltaire

...over and over again.  -- the empirical evidence